Can We Really Trust Big Data?

There has been a lot of discussion about big data. What it is. What it can do, and what it cannot do. Many have written praising articles about how big data will change the way we do business, from predictive analysis to discovering insights hidden in the deluge of unstructured data.

There has been a lot of discussion about big data. What it is. What it can do, and what it cannot do. Many have written praising articles about how big data will change the way we do business, from predictive analysis to discovering insights hidden in the deluge of unstructured data. Others have offered words of caution, citing that data is difficult to manipulate, and simply having a greater volume of data won’t cause the large transformation being hyped about in the media.

As business leaders consider the cost and application of big data, one question they need to ask themselves is, can they really trust big data? After all, plenty of business executives have seen “small” data fail them before, and anytime an analyst makes a recommendation that doesn’t jive with a gut feeling, the first inclination is to question the veracity of the data. It seems that while some companies thrive on data, others approach it with an extreme lack of confidence. This is cause for concern, because a company’s ability to derive valuable insights from big data will depend on its ability to act on those insights with confidence.

On top of an inclination to distrust data, many executives see additional trust concerns with the type of information big data encompassess. Many firms are concerned that big data systems collect huge volumes of unclassified, unanalyzed data that doesn’t undergo the same level of protection, such as encryption, that a traditional database usually employs.

We’ve also heard stories in the media about organizations that have acted on faulty data and suffered because of it. In April 2013, a news organization’s Twitter feed was hacked, and the hacker sent out a false tweet claiming that the White House was under attack. As a result, several investment firms started selling off stocks in what ended up being a huge financial loss for the firms. Stories like this provide a clear wake-up call that data, even big data, isn’t necessarily accurate or beneficial.

Yet this does not mean big data should be ignored. It offers too much potential and solves too many problems to be passed by just because there is a potential of running into bad data. A report from Forrester Consulting found that 61 percent of IT executives felt that increasing their use of business intelligence was either critical or a high priority. Most executives are also interested in the insights they can garner by looking at transactions, structured and unstructured data as a whole group. However, without technology like Hadoop, it is impossible for them to do so. The pressure is also on for faster analysis of more data in order to make better decisions.

So, are companies stuck between a rock and a hard place, or is there a way to trust big data? The real answer is yes, there is, but not all big data solutions are created the same. Hadoop has made huge strides in making extreme-scale data analysis possible but is criticised for not being suitable for enterprise-grade applications. Big data distributors have started creating platforms that overcome the weaknesses of Hadoop by fixing security glitches and making it more accessible to business executives. Some have even moved big data into the world of cloud database. These platforms also provide the add-ons necessary to get real-time analysis, converting Hadoop from an affordable mass storage system to a truly analytical platform.

Most importantly, these platforms provide varying degrees of information governance to create more confidence in the data presented. Those platforms that can test data veracity through data profiling, record-level access control and logging capabilities are those that business executives should be looking into and that will be successful in the long run.

So, yes, figure out how big data can help you, make a plan for how to integrate it and then look for a platform with the tools you need, including a new confidence in what data can do for your company.