3 Big Hadoop Myths Dispelled

As with any new technological innovation, a lot of myths have been generated about Hadoop. At Strata + Hadoop World, held in NY at the end of October, Jack Norris, CMO of MapR, discussed three of those myths and how MapR dispels them.

1. Hadoop Distributions are Incredibly Competitive

Norris declared the vicious competition between Hadoop distributions a myth. There are many commercial Hadoop platforms to choose from, but they all share the same open source code. Since Hadoop was created by open source technology and is in its early stages, it has been necessary for commercial distributors to add their own innovations to the open source code to meet the needs of the enterprise customer. The type of innovations have varied, with some adding management functions on top of the open source product and others adapting the underlying architecture. As a result, there is a diverse ecosystem of Hadoop distributions available that, as the fastest growing big data technology, has been a huge job creator.

2. All NoSQL Solutions are Created Equal

Apache HBase, a NoSQL solution, is integrated with Hadoop and included in every commercial Hadoop distribution. However, just because HBase is included in each distribution does not not mean that it runs the same in each distribution because the architecture that supports HBase varies. A typical architecture for HBase includes multiple layers. HBase will run on Java which runs on HDFS which writes into the Linux file system which writes to the disk. Distributions will vary on how many layers are required to write HDFS, and the more complex those layers are the worse the performance. On the other hand, if those layers are condensed, performance is greatly improved.

3. Hadoop isn’t Enterprise Ready

The most popular criticism of Hadoop is that it isn’t ready to provide real value to enterprises. However, the number of businesses currently having success with Hadoop are more than enough to prove this theory is a myth.

For example, Solutionary analyzes and processes 1 trillion log lines for its security service, and Rubicon Project processes 90 billion ad auctions per day with Hadoop. If you are looking for examples of companies that aren’t Web 2.0, consider a Fortune 100 retailer that runs more than 2000 nodes on Hadoop to use social media to better reach out to its customers or a financial services company that uses Hadoop to mitigate risk and create personalized offers for its customers.

The reality is Hadoop is being used in a variety of industries from Web 2.0 to waste management and healthcare. Currently, the Climate Corporation uses Hadoop to help farmers improve their crop production, and a beverage company in Japan has created kiosks that use facial recognition to create a customized interface.

Hadoop is an incredibly valuable technology that unfortunately is misunderstood by many. While Hadoop distributions vary, especially in their integration of NoSQL, there are enterprise-ready options available that many enterprises are already having success with.