The arrival of “big data” has rejuvenated an industry that had lost its cool. Database connectivity and management software has been around for some time, quietly churning away in the background and allowing financial services companies and other large enterprises to store and use vast quantities of data.
But after the arrival of standards such as ODBC and JDBC and super-sized servers, database management issues faded away – along with media attention on the software providers. Mike Benedict, CEO of database connectivity provider Progress DataDirect, notes that big data has been here for a while, but like the tree that falls in the forest maybe no one was there to hear it.
Today, with the new preponderance of social media data, the vendors are coming back into the limelight. Benedict says: “Data is cool again.”
The next challenge in big data is the ability to monetize it. But there is too much of it, so the ability to collect, interpret and analyze the data is paramount. “You don’t need all of it, you want to pick off the relevant pieces,” says Benedict, “We have to make it accessible and actionable.”
This is what DataDirect’s mission is and its users are demanding access to more and more data sources. “Customers are coming to us to connect them to other sources, they do not want to be locked into one provider,” says Benedict.
DataDirect recently finished working on connectivity to Apache Hadoop, open source software that enables distributed parallel processing of huge amounts of data across servers. DDC has the ability to “bulk load” data into Hadoop, and is now working on being able to connect and pull data out of Hadoop. A preview of the driver was made available in late August.
Connectivity is one piece of the puzzle; monetizing big data will involve both technology solutions AND human interaction, says Benedict. “Technology plus the skill set of data scientists will allow firms to consume, aggregate and take action on big data.”
If there is a downside to big data it is the perception that firms are spying on ordinary people. An application that department store Target used to sense its customers’ needs actually pinpointed the pregnancy of a young woman – who had not yet told her family. Technology companies need to help balance the perception of “big brother” is watching with stories of how big data can help humanity.
“For example,” says Benedict, “If the CDC picks up an outbreak of a disease and stops it before it can harm more people, that’s a good story. We need to balance the Target stories with CDC stories to show how big data can be used in beneficial ways.”
