Big Data Wisdom Courtesy of Monty Python

This article was co-written by Rik Tamm-Daniels, VP Technology for Attivio, Inc. (see Rik’s bio below), and Mike Urbonas.

One of our favorite parts of the hilarious 1975 King Arthur parody, Monty Python and the Holy Grail, is the “Bridge of Death” scene: If a knight answered the bridge keeper’s three questions, he could safely cross the bridge; if not, he would be catapulted into… the Gorge of Eternal Peril!

Unfortunately, that’s exactly what happened to most of King Arthur’s knights, who were either stumped by a surprise trivia question like, “What is the capital of Assyria?” – or responded too indecisively when asked, “What is your favorite color?”

Fortunately when King Arthur was asked, “What is the airspeed velocity of an unladen swallow?” he wisely sought further details: “What do you mean – an African or European swallow?” The stunned bridge keeper said, “I don’t know… AAAGH!” Breaking his own rule, the bridge keeper was thrown over the edge, freeing King Arthur to continue his quest for the Holy Grail.

Many organizations are on “Big Data Holy Grail” quests of their own, looking to deliver game-changing business analytics, only to find themselves in a “boil-the-ocean” Big Data project that “after 24 months of building… has no real value.” Unfortunately, many CIOs and BI Directors have rushed into hasty Hadoop implementations, fueled by a need to ‘respond’ to Big Data and ‘not fall behind.’

That’s just one of the troublesome findings from a recent InformationWeek article by Doug Henschen, Vague Goals Seed Big Data Failures. Henschen’s article cited a recent Infochimps Big Data survey that revealed 55% of big data projects don’t get completed and that many others fall short of their objectives. The top reason for failed Big Data projects was “inaccurate scope”:

Lots of companies are blindly building out Hadoop clusters and collecting new data based on only a vague plan to open up that data store to multiple lines of business in 12 to 24 months…
Dreamers who haven’t thought through their business priorities might think that Hadoop alone will be enough to deliver big data insight, but Infochimp’s study suggests that that’s not the case.

You might say that such CIOs and BI Directors on such an impulsive search for the “Big Data Holy Grail” will inevitably reach the “Bridge of Big Data Reality” (so to speak). And the bridge keeper will then ask, “What business problem are you trying to solve with the help of Big Data?”

Too many IT leaders will have to truthfully answer, “I don’t know… AAAGH!” – and regretfully find themselves in… the dreaded Gartner Big Data Trough of Disillusionment!

The correct response, of course, is to first understand essential details behind the question as King Arthur did. Infochimps CEO Jim Kaskade suggested to InformationWeek a very simple yet “practical and refreshing” question to ask:

Whether it’s churn, anti-money-laundering, risk analysis, lead-generation, marketing spend optimization, cross-sell, up-sell, or supply chain analysis, ask yourself, ‘how many more data elements can you add with big data that can make your analysis more statistically accurate?’ (emphasis added)

The answer to this prudent question will lead to more smart questions:

“What data sources, what types of data, are needed to fulfill my business case – structured data, unstructured data and/or unstructured content?”
“How do I correlate structured and unstructured information together?”
“How do I bring combined data and content into my existing BI infrastructure so users can analyze it using our data visualization tools?”

Big Data success simply cannot be realized without investing the time and effort to deeply understand your business, identify the particular business problem you are trying to solve, and understand the nature of all information your users will need. Doing so is critical to selecting the right technology for your Big Data infrastructure. Now you’re on the track to tackling the Big Data implementation infrastructure conundrum of wisely evaluating countless vendor products and making the right technology investments.

The final joke of Monty Python and the Holy Grail was that it was all a fool’s errand. Similarly, there is no one-size-fits-all Big Data “Holy Grail” technology out there… period. In reality, a successful Big Data architecture consists of multiple components to address the unique aspects of the data. Keep that in mind and show the wisdom of a king by taking pause and asking a few basic business questions to stay on the right path to business-building Big Data success.

The shift in Big Data towards business value creation, and the new technology essential to realize that value, is explored further in Attivio’s new Big Data white paper.

Co-author Rik Tamm-Daniels:

Rik Tamm-Daniels co-founded Attivio and currently serves as Vice President of Technology for Attivio’s Channels and Alliances division developing and executing Attivio’s technical strategy for OEM, SaaS, SI, VAR and Technology Alliance partner recruitment and enablement.