The Big Deal with Big Data

Big data has certainly become big news; today’s headlines are full of tales bemoaning firms’ ability to store, handle and analyze large sets of data. But big data is not a new phenomenon. And finding ways to manage it is not new to software firms, some of which have been doing it for over 20 years.

So why is big data such a hot topic? Jesse Davis, director of research and development at application connectivity provider Progress DataDirect, says Facebook and Twitter are partly at fault.

Big data has certainly become big news; today’s headlines are full of tales bemoaning firms’ ability to store, handle and analyze large sets of data. But big data is not a new phenomenon. And finding ways to manage it is not new to software firms, some of which have been doing it for over 20 years.

So why is big data such a hot topic? Jesse Davis, director of research and development at application connectivity provider Progress DataDirect, says Facebook and Twitter are partly at fault.

He noted that big data became trendy “when the world became aware of data coming from outside the enterprise – i.e. from social media.”

Every keystroke that we make on Facebook or Twitter, every photograph that we enter into a Picasa album, and each time we enable GPS position tracking on our smart phones generates pieces of data. Human beings are the largest contributors to the mountain of big data, says Davis and it is getting bigger by the day.

“Every two days we generate as much data as was generated in all of history up until 2003,” says Davis. “We created 90% of that data in the last two years, and it is growing exponentially.”

Cell phone companies, department stores, online shopping retails all want to mine this data to find patterns they can profit from. They want to see what we are buying, where, and how much we spend. “They pull the data together to get a complete picture of consumers; tying people with places they’ve been and people they know,” says Davis.

If this sounds creepy it isn’t meant to be. Businesses are using big data, much of it generated by us, to try and give us a personalized experience, be it shopping or buying minutes on a cell phone. They are using it to make better business decisions as well. Chevron, for example, uses large sets of data from sonic exploration for oil to analyze and decide where to drill next.

So is big data good for business? It seems so. Davis concludes: “Information gives you a competitive edge, an advantage.”

If big data makes companies more competitive they can give consumers better and more personalized services. Which is a big deal.