Big Data and the Wizard of Oz Syndrome

An excellent article in the Wall Street Journal, “Big Data, Big Blunders,” discussed five

An excellent article in the Wall Street Journal, “Big Data, Big Blunders,” discussed five mistakes commonly made by enterprises when initiating their first Big Data projects. The technology hype cycle, which reminds me a lot of The Wizard of Oz, is a contributing factor in these blunders. I’ll briefly summarize the WSJ’s points, and will suggest, based on my experience helping clients, why enterprises make these blunders.

1. Data for Data’s Sake – Enterprises implement big data for big data’s sake without actually having a business purpose in mind. Almost every widely hyped technology wave has its share of “If we build it, they will come” projects. Technology Project 101: It’s all about the business and for the business. Your first steps are always to get a business sponsor and business objectives.
2. Talent Gap – New technology waves, by definition, lack people with expertise because the technology is so new. With big data analytics in particular, enterprises need data scientists who not only know technology, but possibly one or more of the following: statistics, econometrics, psychology and behavioral science. And that is addition to understanding their enterprise’s business, its competitors and industry. Simply getting tool training, as the article suggests, falls significantly short.
3. Data, Data Everywhere – Enterprises have been busy accumulating data in ever-increasing volumes, variety and velocity. Add in the emerging big data databases, and an enterprises’ ability to gather data explodes. Unfortunately, enterprises have not been as good organizing and understanding the data as they have been gathering it. Big data has no value unless you can understand what you have, analyze it and then act on the insights from the analysis.
4. Infighting – Business people have arguing about business metrics and data ownership as long as enterprises have been collecting data. Business politics, enough said.
5. Aiming Too High – Technologists love to “boil the ocean,” as it’s in our blood. Calm those nerd tendencies and focus on getting real, immediate business value.

Anyone who has been involved in IT projects over the years will find nothing new in these five blunders. Enterprises have been making these blunders in every new technology wave, including analytics, the first wave of data warehouses, dashboards and self-service BI.

Why does history repeat itself? The technology hype cycle is so powerful that enterprises continually stumble in their initial projects for the latest wave.  Often these stumbles are quite costly (and career damaging.) Vendors, analysts and pundits typically pitch the latest technology wave as solving all the problems encountered in the previous cycles. Just as the Wizard of Oz creates the myth of his power (don’t look behind the curtain), so do pundits proclaim the latest technology’s wizardry.  

Google “Big Data” and you will see the magnitude of the media blitz on the subject. Big Data is proclaimed as the data nirvana for enterprises that have been struggling with “old fashioned” data and its associated tools such as relational databases and SQL.  This time it will be different and an enterprise need not worry about information management.

Big data will likely be similar to the initial web wave: the Internet Bubble. The web did change the world as the people hyping it at the time proclaimed, but not before many enterprise failures and vendor bankruptcies. It took much failure and more hype than the pundits thought, and it is likely Big Data will follow some of the same path.