Relying on Data Can Lead to the Wrong Decisions Says CFO.com

I just read a new CFO Magazine article called “That New Big Data Magic”, which contains the following opening paragraphs:

Back in the 1980s, American Airlines (AA) was partnered with British Airways (BA), and AA’s marketing head wanted to know how many of the airline’s gold card members, its most profitable customers, were flying BA rather than AA. Larry Tieman, then a managing director in IT at AA, dove into the massive amounts of data AA had collected on its customers and reported back that, basically, all of AA’s gold card holders were flying BA. The reason seemed obvious: BA’s service simply was better. So AA launched a huge quality-improvement initiative. It upgraded its food service and changed its flight schedules to increase on-time rates. It invested heavily to compete on quality on all fronts.
The results, Tieman recalls, were disastrous.

“None of it mattered,” says Tieman, who between 2000 and 2010 was a senior technology vice president at FedEx. “The quality of food meant nothing. What was sticky was frequent flyer loyalty programs.”
“With Big Data,” Tieman says, “you may be spot-on about a problem, but the solution doesn’t magically appear out of the data.”

The story goes on to say:

“Relying on data alone could lead a company down the path AA took: investing heavily in the wrong things.”

I re-read this several times, and still couldn’t make any sense out of the anecdote: if frequent flyer loyalty was key, why didn’t the AA fliers stick with AA flights to get the miles? I assume the partnership gave them air miles when they used BA – but presumably they could still get miles on AA, too, so the part about “what was sticky was the frequent flier program” is true but irrelevant and we’re back to the original question, which remains unanswered: why did they fly BA rather than AA?

The anecdote doesn’t back up the conclusion, leading to lots of other questions:

Did flying BA give them more miles than BA through the partnership? (sounds like a strange partnership agreement)
Were the people flying BA only there was no AA flight available? (marketing head wanted to know when people were flying BA rather than AA, so this would be bad analysis)
Were the BA flights free ones, using up the air miles? (even so, what difference would it make to the choice equation?)
Did the gold card fliers also have BA cards, and these trumped their AA cards? (I know they used to offer gold cards to people who already had gold cards on other airlines, with the logic that these were good customers, that they wouldn’t change carriers without similar conditions, and the marginal cost of extra people in the priority lineup, etc. is very low)
Was the routing different/better (this would explain why food service and schedules didn’t make any difference, but would change the conclusion of what was important)…

In order to find out what was going on, I swapped emails with Larry Tieman, whose quote in the article explains why it’s wrong to “rely on data alone” (you need people):

“what you do with [data] is a people-based activity, a skill base you have to mature. And it doesn’t come quickly.”

It turns out that frequent fliers did indeed get air miles on either airline, and that the “obvious reason” was also the real one: on head-to-head flights, frequent fliers did indeed choose BA because of “the quality of business and first class”. The “quality from the ground up” initiative mentioned cost $1Bn, with little impact: as the last two decades have shown, what most fliers are interested in is lower prices.

So, unlike the article’s assertion, AA’s problems don’t seem to have had anything to do with “relying on data alone”. Larry’s conclusion over email was rather different:

“My moral, the data can tell you what is wrong but fixing it is another thing all together.”

Overall, it’s maybe a good example of what plagues most business analysis — the data is incomplete and confusing, and isn’t necessarily related to the most important problems people have to deal with! The rest of the article is less controversial, and include a nice summary of what CFOs (and everybody else) should be thinking about in analytics:

“…new technologies are making it more economical to make sense of Big Data which, in fact, has been around for a long, long time. The caveat is that those technologies will not provide those opportunities. That’s still up to the people who make business decisions.”

Finally, as a footnote, I’m with Andy Bitterer that there are four ‘V’s to the information challenge – not just Volume, Velocity, and Variety, but also Validity…