Analytics and Ending the Tyranny of the Anecdotal

I was chatting with our VP of business development today and she shared a great phrase that she had heard recently – “the tyranny of the anecdotal.” This refers, of course, to the terrible tendency of people to use anecdotes and stories to make decisions.

I was chatting with our VP of business development today and she shared a great phrase that she had heard recently – “the tyranny of the anecdotal.” This refers, of course, to the terrible tendency of people to use anecdotes and stories to make decisions. The regrettable tendency for a doctor to ignore data about other people’s patients in favor of his or her personal experience with patients, for a parent to think that a school is a good school because their kids did well there or for political opinion to be presented as divided because the radio had two stories, one on each side of some question. While this kind of thing may be irritating in our personal lives, it is a serious issue in running a business.

Think about it. What we are talking about is making really important decisions not based on facts or analysis of facts but on the subjective stories and experiences of a few. Given that a decision is (according to MW) “a determination arrived at after consideration” this seems like a real problem. Making decisions based on anecdotes, stories is something that goes on in businesses of every type and at every level. Call center representatives treat customers a certain way because it has worked with their customers before, people pick suppliers because someone else says they are reliable, tellers promote a bank account type based on conversations over lunch and so on. But this use of stories does not consider the data, does not segment customers or suppliers before deciding that what works with one will work with another and does not provide any kind of comparison testing between approaches. So what can you do to avoid the “tyranny of the anecdotal” in your business?

1. Use analytics to drive decision-making
   Using analytic techniques, especially data mining and predictive analytics, to analyze historical data to see what really works and for whom drives better decision-making than listening to a few anecdotes (even your own). This is particularly true of the kind of high volume operational decisions that are everywhere in your business as these generate lots of data for analysis.
2. Experiment
   Use A/B testing or champion/challenger testing to make sure that what you think will work actually works. Running experiments to compare different approaches helps you ensure that what you learned from a set of data is what you should have learned from it.
3. Have a control group
   Never do anything new 100% of the time – always keep a control group. Having a control let’s you see if what you have come up with is actually better than doing nothing, the business equivalent of testing a drug against a placebo.
4. Decision analysis
   Tying decisions made and decision-making approaches to business outcomes let’s you see what really works, in business terms, and keeps your analytics focused not on being “predictive” or “statistically accurate” but on adding business value.

Decision Management is an explicit rejection of the tyranny of the anecdotal – it replaces uncontrolled, and largely uncontrollable decision-making outside a system with Decision Management Systems that use analytics, experimentation, control groups and decision analysis to find out what really works.