By using this site, you agree to the Privacy Policy and Terms of Use.
Accept
SmartData Collective
  • Analytics
    AnalyticsShow More
    data analytics in sports industry
    Here’s How Data Analytics In Sports Is Changing The Game
    6 Min Read
    data analytics on nursing career
    Advances in Data Analytics Are Rapidly Transforming Nursing
    8 Min Read
    data analytics reveals the benefits of MBA
    Data Analytics Technology Proves Benefits of an MBA
    9 Min Read
    data-driven image seo
    Data Analytics Helps Marketers Substantially Boost Image SEO
    8 Min Read
    construction analytics
    5 Benefits of Analytics to Manage Commercial Construction
    5 Min Read
  • Big Data
  • BI
  • Exclusive
  • IT
  • Marketing
  • Software
Search
© 2008-23 SmartData Collective. All Rights Reserved.
Reading: 3 Harsh Truths about Big Data
Share
Notification Show More
Latest News
data analytics in sports industry
Here’s How Data Analytics In Sports Is Changing The Game
Big Data
data analytics on nursing career
Advances in Data Analytics Are Rapidly Transforming Nursing
Analytics
data analytics reveals the benefits of MBA
Data Analytics Technology Proves Benefits of an MBA
Analytics
anti-spoofing tips
Anti-Spoofing is Crucial for Data-Driven Businesses
Security
ai in software development
3 AI-Based Strategies to Develop Software in Uncertain Times
Software
Aa
SmartData Collective
Aa
Search
  • About
  • Help
  • Privacy
Follow US
© 2008-23 SmartData Collective. All Rights Reserved.
SmartData Collective > Big Data > 3 Harsh Truths about Big Data
Big Data

3 Harsh Truths about Big Data

Eran Levy
Last updated: 2016/10/06 at 1:14 PM
Eran Levy
7 Min Read
SHARE

Takeaways from Strata NYC

With this year’s Strata + Hadoop World conference in New York over, and after recovering from a hectic week of live demos, presentations and introducing Sisense to anyone who had half a mind to listen, I wanted to take a moment to look back at the event and mostly – how it reflects on the current big data landscape.

Contents
Takeaways from Strata NYCTakeaways from Strata NYCBig Data is Still Really, Really HardWhere Has the Visualization Crowd Gone?Back to Basics: Big Data is all about Data Prep

Takeaways from Strata NYC

With this year’s Strata + Hadoop World conference in New York over, and after recovering from a hectic week of live demos, presentations and introducing Sisense to anyone who had half a mind to listen, I wanted to take a moment to look back at the event and mostly – how it reflects on the current big data landscape.

Because before everything else, Strata is the place where people go to talk about bona fide big data: not the marketing-speak, amorphous “big data” that vendors use to describe anything with a whiff of interactive charts or a glimmer of NoSQL; but the actual, massive, unstructured or semi-structured datasets that only a minority of businesses are dealing with today.

More Read

data analytics in sports industry

Here’s How Data Analytics In Sports Is Changing The Game

Advances in Data Analytics Are Rapidly Transforming Nursing
Data Analytics Helps Marketers Substantially Boost Image SEO
Data Visualization Boosts Business Scalability with Sales Mapping
Data-Driven Marketing Offers Huge Benefits for Landscapers

Adi Azaria at Strata + Hadoop, NYC 2016
Our booth at Strata

These organizations come to Strata to talk about ways to tame this data beast in a business context. But oddly enough the conversation was mostly about storage, processing, and clustering – topics which are certainly interesting but are still a far cry from tangible business benefits. This relates to the first point I want to make, which is that all hype aside, big data is still massively complex and far from straightforward to work with.

Big Data is Still Really, Really Hard

One of the most immediately noticeable things about Strata is that this is a conference for technical people. Like, super-technical, with keynotes such as File format benchmark: Avro, JSON, ORC, and Parquet. Again, there’s nothing wrong with this, I’m a data geek myself and I actually learned a lot. But it does hint at a larger phenomenon, namely: big data is still from far simple, and the discussion around it remains technical in nature and dominated by technical jargon.

With the amount of time big data has been around, one might have expected the questions around it be more business-oriented; but the truth is, there’s still a ton of complexity involved when you’re working with huge datasets from disparate sources. So much time was devoted to discussions about storage mechanisms for streaming data or log files, that it seemed to beg the question: what is the point of all this? Why do we need to invest all these resources into capturing the data, when most of the vendors failed to offer any kind of solution for querying and analysis – i.e., the stuff that generates the aforementioned tangible business value?

In fact, the usual suspects – the modern data discovery / BI crowd – was surprisingly absent. Except for Sisense and a handful of other companies, most business intelligence vendors weren’t at Strata this year. Which leads me to my next point: the market is realizing the data visualization tools aren’t, and never will be, a solution for big data analytics

Where Has the Visualization Crowd Gone?

In-memory data visualization tools have been all the rage for the past decade or so, having proven their ability to accelerate analytics and to “democratize” data by making it accessible to business departments with limited technical resources.

However, while these tools are amazing at what they do, they also have limitations: they’re not meant for complex data. They work best with simple data, or data that has been aggregated and summarized beforehand so that it can neatly fit into a local or cloud server’s RAM; but trying to analyze actual big data with an in-memory BI tool is an exercise in futility. Eventually, you’ll find yourself investing in more hardware, third party software, or accepting that you have to look at just a subset of the data (we’ve written about this many times before – for example here and here).

So really it’s no surprise that we didn’t see the visualization vendors at Strata: because if you actually have big data, these tools won’t solve your problem. At best they’ll give you a nicer visualization layer to use on top of whatever technology you’ll use as your back-end for storing, modeling and wrangling the data.

Back to Basics: Big Data is all about Data Prep

While the BI crowd wasn’t majorly present, we did see a lot of machine learning – both on the floor and in the presentations and keynotes. Most of the more interesting technologies on display were offering various automated or semi-automated data preparation, and there were a ton of predictive algorithms for solving verticalized issues (e.g. churn analysis and event scoring).

And herein lies the final insight I got from Strata – which is that after more than 10 years of hype, data preparation is still the absolute most important factor when it comes to big data. That’s why the modern data discovery vendors weren’t there; that’s why Sisense was. The ability to make sense of big data still lies in ETL, smart data modeling, and efficient utilization of computational resources – issues that the modern BI vendors have all but abandoned in favor of one more type of graph or some nicer looking pie charts.

Strata was amazing for us data geeks; but before big data can actually fulfill its promise and become the mainstream technology it was already supposed to be, the data preparation processes involved with it will have to be simplified – and the tools that use natural language processing. machine learning, deep learning, bots and other new technologies will all play a role in this.

Eran Levy October 6, 2016
Share this Article
Facebook Twitter Pinterest LinkedIn
Share

Follow us on Facebook

Latest News

data analytics in sports industry
Here’s How Data Analytics In Sports Is Changing The Game
Big Data
data analytics on nursing career
Advances in Data Analytics Are Rapidly Transforming Nursing
Analytics
data analytics reveals the benefits of MBA
Data Analytics Technology Proves Benefits of an MBA
Analytics
anti-spoofing tips
Anti-Spoofing is Crucial for Data-Driven Businesses
Security

Stay Connected

1.2k Followers Like
33.7k Followers Follow
222 Followers Pin

You Might also Like

data analytics in sports industry
Big Data

Here’s How Data Analytics In Sports Is Changing The Game

6 Min Read
data analytics on nursing career
Analytics

Advances in Data Analytics Are Rapidly Transforming Nursing

8 Min Read
data-driven image seo
Analytics

Data Analytics Helps Marketers Substantially Boost Image SEO

8 Min Read
data visualization for small business
Data Visualization

Data Visualization Boosts Business Scalability with Sales Mapping

7 Min Read

SmartData Collective is one of the largest & trusted community covering technical content about Big Data, BI, Cloud, Analytics, Artificial Intelligence, IoT & more.

giveaway chatbots
How To Get An Award Winning Giveaway Bot
Big Data Chatbots Exclusive
ai is improving the safety of cars
From Bolts to Bots: How AI Is Fortifying the Automotive Industry
Artificial Intelligence

Quick Link

  • About
  • Contact
  • Privacy
Follow US

© 2008-23 SmartData Collective. All Rights Reserved.

Removed from reading list

Undo
Go to mobile version
Welcome Back!

Sign in to your account

Lost your password?