Taming Big Data Is Not a Technology Issue

November 13, 2012
47 Views

One thing that has struck me recently is that most of the focus when discussing big data is upon the technologies involved. The consensus seems to be that the biggest challenge with big data is a technological one, yet I don’t believe this to be the case.

One thing that has struck me recently is that most of the focus when discussing big data is upon the technologies involved. The consensus seems to be that the biggest challenge with big data is a technological one, yet I don’t believe this to be the case. Sure, there are challenges today for organizations using big data, but, I would like to submit to you that technology is not the biggest problem. In fact, technology may be one of the easiest problems to solve when it comes time to tame big data.

The fact is that there are tools and technologies out there that can handle virtually all of the big data needs of the vast majority of organizations. As of today, you can find products and solutions that do whatever you need to do with big data. Technology itself is not the problem.

Then, what are the issues? The real problems are with resource availability, skills, process change, politics, and culture. While the technologies to solve your problems may be out there just waiting for you to implement them, it isn’t quite that easy, is it? You have to get budget, you have to do an implementation, you have to get your people up to speed on how to use the tools, you have to get buy in from various stakeholders, and you have to push against a culture averse to change.

The technology is right there, but you are unable to effectively put it to work. It FEELS like a technology issue since technology is front and center. However, it is really the cultural, people, and political issues surrounding the technology that are the problem. Let me illustrate with an example.

A few weeks back, I was at the annual Teradata Partners conference having a really good discussion with someone from one of our larger customers. When I suggested to him that his struggle with big data isn’t a technology problem, he challenged me. He pointed out that the network protocols they have in place just can’t support the level of data flow required to handle the big data they need to utilize. He felt he had a very real technological issue.

I pushed back on him and asked if there weren’t widely available products that would solve his network bandwidth problem tomorrow if he were to implement them. He said that was true but that he couldn’t just go and implement them tomorrow. I then pointed out to him that he had just proven he didn’t have a technology problem. The technology exists. He knows what it is and how to put it in place. He just can’t get it in his hands.

His real problem is with a culture that doesn’t want to invest in upgrading the network at this time. His real problem is a bureaucratic budgeting process that takes a long time to navigate. His real problem is building a business case to prove the need and value of making the upgrade. All of these problems are very real for my client and are also very valid. But, they aren’t about technology. Brush aside corporate policies, give him cash, and give him purchase authority today, and his problem will be solved very quickly.

I suspect that as many readers consider the story above, they will realize that they don’t really have a technology problem either. Technology is the easy part. It exists and it can be put in place as soon as you’re ready. However, those pesky budgets, policies, skill gaps, and cultural realities get in the way of that happening.

As your organization proceeds with the process of incorporating big data, be sure to give appropriate attention to the non-technical hurdles that you’ll face along the way. Don’t mistake them for a technology problem and don’t underestimate them. These hurdles will often be much more difficult to clear than finding a set of tools and technologies that will do what you require.

A few things to keep top of mind:

  1. Think through the challenges you’ll face in getting your organization to effectively use the big data tools you’ve pushed so hard to acquire.
  2. Plan to help your organization embrace the business model and process changes that might be suggested by results that your new tools and analytics uncover within the new data sources.

The fact that the tools and technologies to tame big data are readily available today is a terrific thing and you need not lose sleep on that front. However, there may be a lot of sleep loss as you figure out how to get your organization to actually acquire, implement, and put the tools and technologies to use. The people, cultural, and political issues surrounding big data are very big and very real.

To see a video version of this blog, visit my YouTube channel.

Originally published by the International Institute for Analytics

 

You may be interested

IEEE Big Data Conference 2017 to Highlight Challenges, Opportunities
Big Data
65 shares967 views
Big Data
65 shares967 views

IEEE Big Data Conference 2017 to Highlight Challenges, Opportunities

Ryan Kade - June 23, 2017

Since 2013, the Institute of Electrical and Electronics Engineers has held annual big data conferences to highlight changes and opportunities…

10 of the Top Marketing BI Software Options
Business Intelligence
117 shares1,425 views
Business Intelligence
117 shares1,425 views

10 of the Top Marketing BI Software Options

Hayden B. - June 23, 2017

Business can be complicated sometimes. It’s not always easy to keep track of all the data and information we deal…

The Race for 5G Is the Race for Data Dominance
Big Data
80 shares1,120 views
Big Data
80 shares1,120 views

The Race for 5G Is the Race for Data Dominance

Daniel Matthews - June 22, 2017

Have you noticed how often the phrase “by the year 2020” comes up? In the tech sphere, many are heralding…