Changing Rules: Empirical vs Anecdotal Evidence

Everyone wants a better game. GMs will do many things to achieve this goal. Unfortunately, GMs delve into the role of game designer way too quickly without understanding a simple truth: GMs are not game designers. The two have different objectives. GMs want a fun game with a certain tone. Game designers try to build a system that is fun, but not vulnerable to exploitation and imbalance.

Normally, this is not a big deal. A GM can use experience to make small tweaks to a system to the get the game they want. If it works well, they want to pick it up and use it everywhere. Doing this can often cause more harm than good. As a GM and designer who tweaks my own system, I understand the desire and the dangers and have dealt with them first hand. The biggest alligator lurking in that swamp is the mystery of empirical evidence vs anecdotal evidence.

Evidence is data that contributes to drawing a conclusion. In this case we are talking about evidence for changing / adding / ignoring a rule. When we collect data, usually through actually playing the game, we need to understand that our data is not perfect. Every player carries baggage with them from other experiences, other games. We also do not play our games perfectly. Each person involved has their own state of mind, influenced by the rest of their life, which may or may not be conducive to a good game. Relationships between players may cloud the objectives of play. Our observations during play have this "noise" from all of the temporary conditions associated with the specific instance of the game and game group.

Ultimately this noise means that one observation is likely to be flawed in some way. The only way to get rid of this "noise" is to make many observations over may instances. Taken together these observations allow us to look at how things work as a whole as opposed to a single or few flawed observations. These many observations taken together form significant empirical evidence. 

A single or few flawed observations are called anecdotal evidence. Unfortunately this type of evidence is the most relatable and leads to statements like "It worked fine in my game." and "We never has a problem with it."  Anecdotal evidence, not surprisingly, often comes along with a story of how it worked or didn't work on one occasion. However, because it is all anecdotal, it might not apply to your game. One observation of success might not even apply to the next scenario in the same group.

When making rule changes, it is important to understand the difference between solid empirical evidence and anecdotal evidence. You might have your gaming group that you've played with for years. Based on a single game, you might decide to change a rule. You change it and everything works well. This change based on anecdotal evidence works well because of your group. Pick that same rule up and move it to another group, and it won't work at all. 

The only way to gather solid empirical evidence for game design is playing over a wide variety of groups, with different people, and in different scenarios. This is what a game designers needs to develop a bullet-proof rule set that is impervious to exploitation, min-maxing, and general player mayhem. This is what WoTC did with 5E playtesting, and it clearly shows. And even after thousands upon thousands of playtests, the outcome still wasn't perfect. Achieving perfection in a ruleset is hard. 

I'm not advocating extensive playtesting of every rule change. However, when basing changes on anecdotal evidence, be aware that what works in one group isn't going to work in another, that what works in this scenario isn't going to work in another, that what one player uses well another player will exploit. Share your ideas and lets other consider them, but never assume that your gaming group or your one game session is the gold standard for making rules changes. The world is bigger than that.