Big Data After the Hype

January 7, 2013
59 Views

Total Data Management

This year, I’ve been following the meteoric rise of big data.

Total Data Management

This year, I’ve been following the meteoric rise of big data. It has been a boon for vendors who are venturing into this area.  It has produced countless start-ups and much buzz in the data management world.

However, when it comes down to it, what we’re really talking about here is data management and data governance.  Whether you have to deal with big data, enterprise data or spread-marts, data needs to be managed no matter what size. The tides are turning for a total data management approach. Recent surveys shows that despite the market hype, most technologists and business users feel that big data is an off-shoot of data management, not a branch of technology in itself. 

So, why the hype?  I’m convinced it is mostly vendor-generated. In 2010, when big data began to gain notoriety, there was a disconnect for some vendors.  While partnered with traditional enterprise data management companies like the Oracle and IBM’s of this world, not all vendors were prepared for the growing popularity of open source and Hadoop. Others were (and still are) better positioned. They began talking about big data as a product differentiator. Vendors who don’t have the basic architecture for managing data in Hadoop have been and will continue to struggle. 

For example, ETL tools that have a basic connection to move data in and out of Hbase, Hortonworks and Cloudera can’t stop there.  The power of Hadoop must be harnessed, and it’s not always an easy thing to do when your technology requires executables tied to CPUs.  One of the powerful things about Hadoop is that it scales based on a languages like PIG, Sqoop and Java without having to install anything.  Want to expand the number of servers?  Add a datanode server, tell the name namenode and rebalance – and your off and running.  However, even this simple innovation is more difficult on some vendors’ architectures than others.

Another rethinking that is taking place in the market is long-standing CPU-based pricing structure.  Vendors who they keep their pricing structure based on core processors for Hadoop will continue to struggle because it runs counter to the power of Hadoop. You hear about the volume, velocity and variety.  Technically, if you want to step up the volume with another datanode, it’s no big deal. However, it becomes a big deal if you have to renegotiate a vendor contract each time.

Last year, around this time, I did write about the various costs associated with the scale of data. In summary, the costs of licenses and connectors are the bigger for enterprise data, while the costs associated with skills are more likely to affect you with big data.  There will com a time where the skills gap will be closed, however.

In the year 2013, we’ll begin to see the un-hyping of big data in favor of this total data management approach. For buyers, big data will be a tick-box in their RFP’s in the effort to manage data, no matter what the size.

You may be interested

IEEE Big Data Conference 2017 to Highlight Challenges, Opportunities
Big Data
65 shares1,030 views
Big Data
65 shares1,030 views

IEEE Big Data Conference 2017 to Highlight Challenges, Opportunities

Ryan Kade - June 23, 2017

Since 2013, the Institute of Electrical and Electronics Engineers has held annual big data conferences to highlight changes and opportunities…

10 of the Top Marketing BI Software Options
Business Intelligence
117 shares1,501 views
Business Intelligence
117 shares1,501 views

10 of the Top Marketing BI Software Options

Hayden B. - June 23, 2017

Business can be complicated sometimes. It’s not always easy to keep track of all the data and information we deal…

The Race for 5G Is the Race for Data Dominance
Big Data
80 shares1,163 views
Big Data
80 shares1,163 views

The Race for 5G Is the Race for Data Dominance

Daniel Matthews - June 22, 2017

Have you noticed how often the phrase “by the year 2020” comes up? In the tech sphere, many are heralding…