Raise the Big Data Flim-Flam High

If there were ever a more apt rallying slogan for the Big Data BS babblers it would be “We BS about Big Data so that you don’t need to think”… and you know what? That’s how it is working.

The trouble with the hype is that almost everyone and their dog is in on it. From the freelance or indentured Big Data gurus to the Gartners, IBMs and HPs of this world. Everyone who is anyone is trying to jump on the Big Data bandwagon, whether it makes sense or not. Hell, if I could become ludicrously rich and infamous on the back of Big Data, I would jump the Big Data shark as well.

The other trouble is that the Big Data hype is very inconsistent in almost all areas, apart from the general unstated agreement there seems to be that Big Data will bring riches beyond the dreams of avarice, for everyone who wants it.

So, let’s assuming that one wants to cash in on Big Data, what’s the first thing that we need to understand?

Big Data comes in big data volumes, it has many data varieties, meaning it has a number of distinct formats, and it comes at us with increasing velocity, which most of the time we simply do not notice.

So what does that tell us? Right, Big Data is data; just more of it, more flavours of it, generated and transmitted at faster and faster rates.  To simplify, data is like water (Oh, no not another analogy) and whereas Data Warehousing is the Rhine or the Mississippi Delta, Big Data is the Ma and Pa of the Iguazu, Victoria and Niagara Falls.

So, what was the next questions I asked myself on the way to the land of Big Data health, wealth and happiness?

I asked, “if you have a Big Data success story then let’s hear the skinny”, such as:

• Please detail data that has used to create new insight and understanding?
• How was this data sourced, treated and stored?
• How was the resulting data queried? Let’s see the queries, the code, the pseudo-code and the code narrative.
• What were the results of the queries? In technical and business terms, please.
• What normalisation of the results took place?
• How did those results drive insight? In business terms, please.

Perfectly straightforward, right? These are the sorts of questions that one should be able to ask of a Data Warehouse user and then reasonable expect to get a coherent set of answer back in return.

Well, seems like when it comes to Big Data this sort of line of questioning and reasoning is somewhat problematic.

Which is a problem, because I am seeing fantastic claims made for Big Data, which is great, and I wish we all could become more prosperous from Big Data, but I can’t seem to get a handle on quite how one goes about it. It’s as if ‘tangible’ was anathema when it comes to practical and detailed examples of Big Data in action. You know, driving tangible business benefits through an understandable value chain of processes, ingredients and outcomes.

So, come on Big Data guys, gals and gurus, it’s time now to pony up, put up or shut up.