3 Harsh Truths about Big Data

Takeaways from Strata NYC

With this year’s Strata + Hadoop World conference in New York over, and after recovering from a hectic week of live demos, presentations and introducing Sisense to anyone who had half a mind to listen, I wanted to take a moment to look back at the event and mostly – how it reflects on the current big data landscape.

Takeaways from Strata NYC

With this year’s Strata + Hadoop World conference in New York over, and after recovering from a hectic week of live demos, presentations and introducing Sisense to anyone who had half a mind to listen, I wanted to take a moment to look back at the event and mostly – how it reflects on the current big data landscape.

Because before everything else, Strata is the place where people go to talk about bona fide big data: not the marketing-speak, amorphous “big data” that vendors use to describe anything with a whiff of interactive charts or a glimmer of NoSQL; but the actual, massive, unstructured or semi-structured datasets that only a minority of businesses are dealing with today.

These organizations come to Strata to talk about ways to tame this data beast in a business context. But oddly enough the conversation was mostly about storage, processing, and clustering – topics which are certainly interesting but are still a far cry from tangible business benefits. This relates to the first point I want to make, which is that all hype aside, big data is still massively complex and far from straightforward to work with.

Big Data is Still Really, Really Hard

One of the most immediately noticeable things about Strata is that this is a conference for technical people. Like, super-technical, with keynotes such as File format benchmark: Avro, JSON, ORC, and Parquet. Again, there’s nothing wrong with this, I’m a data geek myself and I actually learned a lot. But it does hint at a larger phenomenon, namely: big data is still from far simple, and the discussion around it remains technical in nature and dominated by technical jargon.

With the amount of time big data has been around, one might have expected the questions around it be more business-oriented; but the truth is, there’s still a ton of complexity involved when you’re working with huge datasets from disparate sources. So much time was devoted to discussions about storage mechanisms for streaming data or log files, that it seemed to beg the question: what is the point of all this? Why do we need to invest all these resources into capturing the data, when most of the vendors failed to offer any kind of solution for querying and analysis – i.e., the stuff that generates the aforementioned tangible business value?

In fact, the usual suspects – the modern data discovery / BI crowd – was surprisingly absent. Except for Sisense and a handful of other companies, most business intelligence vendors weren’t at Strata this year. Which leads me to my next point: the market is realizing the data visualization tools aren’t, and never will be, a solution for big data analytics

Where Has the Visualization Crowd Gone?

In-memory data visualization tools have been all the rage for the past decade or so, having proven their ability to accelerate analytics and to “democratize” data by making it accessible to business departments with limited technical resources.

However, while these tools are amazing at what they do, they also have limitations: they’re not meant for complex data. They work best with simple data, or data that has been aggregated and summarized beforehand so that it can neatly fit into a local or cloud server’s RAM; but trying to analyze actual big data with an in-memory BI tool is an exercise in futility. Eventually, you’ll find yourself investing in more hardware, third party software, or accepting that you have to look at just a subset of the data (we’ve written about this many times before – for example here and here).

So really it’s no surprise that we didn’t see the visualization vendors at Strata: because if you actually have big data, these tools won’t solve your problem. At best they’ll give you a nicer visualization layer to use on top of whatever technology you’ll use as your back-end for storing, modeling and wrangling the data.

Back to Basics: Big Data is all about Data Prep

While the BI crowd wasn’t majorly present, we did see a lot of machine learning – both on the floor and in the presentations and keynotes. Most of the more interesting technologies on display were offering various automated or semi-automated data preparation, and there were a ton of predictive algorithms for solving verticalized issues (e.g. churn analysis and event scoring).

And herein lies the final insight I got from Strata – which is that after more than 10 years of hype, data preparation is still the absolute most important factor when it comes to big data. That’s why the modern data discovery vendors weren’t there; that’s why Sisense was. The ability to make sense of big data still lies in ETL, smart data modeling, and efficient utilization of computational resources – issues that the modern BI vendors have all but abandoned in favor of one more type of graph or some nicer looking pie charts.

Strata was amazing for us data geeks; but before big data can actually fulfill its promise and become the mainstream technology it was already supposed to be, the data preparation processes involved with it will have to be simplified – and the tools that use natural language processing. machine learning, deep learning, bots and other new technologies will all play a role in this.