Hadoop Adoption: Where Does Your Company Stand?

Any discussion about big data analytics will likely bring up the topic of Hadoop. Hadoop is certainly not a new platform and can effectively change the way businesses utilize their big data, but despite its advantages, Hadoop adoption remains largely lacking. Put in simple terms, Hadoop is an open source software framework that helps store, organize, and analyze data distributed across multiple clusters and nodes. Businesses that use it find that they’re able to work with more data than they ever have before, along with a wider variety of data. Even so, many organizations remain unconvinced or simply see Hadoop as not necessary for what they’re doing. Your company may feel the same way. It’s a barrier that Hadoop supporters have been trying to overcome for several years now, and success in that realm has been fleeting at best.

Let’s look at some of the recent numbers showing lack of Hadoop adoption. A recent survey from Gartner shows that 54 percent of IT and business leaders said they had no plans of investing in Hadoop within the next two years. While that still means nearly half do have plans of some sort, for a revolutionary technology, less than half seems awfully low. Yes, Hadoop adoption continues to happen, but as Gartner characterizes it, that adoption is “steady” but “slow.” Businesses are coming around to deploying Hadoop, but the rate remains well below what it seems like it should be. This is a trend that can be frustrating to Hadoop proponents. For all the good Hadoop can do, too many organizations refuse to hop on board the bandwagon.

If your company is one of those resistant to adopting Hadoop, you may have perfectly good reasons for not doing so. Gartner’s survey also sheds some light on the justifications for avoiding Hadoop for the time being. Many businesses stated that they were having a difficult time trying to accurately express the business value Hadoop actually provided. Big data’s benefits and advantages are usually well understood, but Hadoop remains a bit more complicated to explain. Put bluntly, organizations don’t feel like they actually need it at this time.

Even more concerning is the fact that 57 percent of survey respondents said the skills gap was one of the biggest challenges to using Hadoop. The platform is complex and often requires extensive experience with programming and data science to utilize it properly. Many businesses have found that they simply don’t have the employees with the skills to make Hadoop work for them. This is just a snapshot of an even bigger problem most organizations are encountering when it comes to big data — there aren’t enough people with the talents to perform analytics. There are, of course, steps being taken to educate a new generation of data scientists, but it’s still a massive game of catch-up as demand greatly outpaces supply.

One other reason Hadoop adoption may appear lacking has to do with timing. Many new products and technologies follow Gartner’s “hype cycle,” with Hadoop having spent most of its time in the early adopter phase. However, we may be moving out of that phase and transitioning into more mainstream success. Such transitions do make for some rough spots, and while Hadoop develops into more of a tool accepted by the mainstream, adoption will remain sluggish. Only once Hadoop has cleared this hurdle will we likely see Hadoop adoption pick up.

There are a number of ways to see that mainstream acceptance happen sooner rather than later. Large tech companies and vendors will first adopt Hadoop for their own tools and platforms, something we’re already seeing right now from the likes to Microsoft, Amazon Web Services, and IBM. More tools based upon SQL, such as Apache Pig will also lead to a greater understanding of what Hadoop can do. Most of these steps simply involve educating businesses on how Hadoop can be used to benefit them. Once this is understood, adoption will likely increase.

In many ways Hadoop is like other growing technologies like cloud computing and what is flash storage. It will take a period of time before businesses grow comfortable using it, but once they do, the possibilities are nearly endless. Organizations will be in position to truly take full advantage of big data. Hadoop will be looked at as a key component in that process. We’re likely only a few years away from that happening.