The Future of Big Data: Good, Bad or Ugly?

The Pew Internet Project, in collaboration with Elon University’s Imagining the Internet Center, recently published a report on the future of big data, based on a “non-random online sample of 1,021 Internet experts and other Internet users.”

The Pew Internet Project, in collaboration with Elon University’s Imagining the Internet Center, recently published a report on the future of big data, based on a “non-random online sample of 1,021 Internet experts and other Internet users.”

Fifty-three percent of the survey respondents agreed with the positive scenario presented to them: “Thanks to many changes, including the building of ‘the Internet of Things,’ human and machine analysis of large data sets will improve social, political, and economic intelligence by 2020. The rise of what is known as ‘Big Data’ will facilitate things like ‘nowcasting’ (real-time ‘forecasting’ of events); the development of ‘inferential software’ that assesses data patterns to project outcomes; and the creation of algorithms for advanced correlations that enable new understanding of the world.

Overall, the rise of Big Data is a huge positive for society in nearly all respects.” The report includes many specific quotes to support this view. For example, this one from Tiffany Shlain, director and producer of the film Connected and founder of The Webby Awards: “Big Data allows us to see patterns we have never seen before. This will clearly show us interdependence and connections that will lead to a new way of looking at everything. It will let us see the ‘real-time’ cause and effect of our actions. What we buy, eat, donate, and throw away will be visual in a real-time map to see the ripple effect of our actions. That could only lead to mores-conscious behavior.”

Not all survey respondents were so enthusiastic about big data. Thirty-nine percent agreed with the negative scenario presented to participants: “Thanks to many changes, including the building of ‘the Internet of Things,’ human and machine analysis of Big Data will cause more problems than it solves by 2020. The existence of huge data sets for analysis will engender false confidence in our predictive powers and will lead many to make significant and hurtful mistakes. Moreover, analysis of Big Data will be misused by powerful people and institutions with selfish agendas who manipulate findings to make the case for what they want. And the advent of Big Data has a harmful impact because it serves the majority (at times inaccurately) while diminishing the minority and ignoring important outliers. Overall, the rise of Big Data is a big negative for society in nearly all respects.” Here’s a sample quote from Marcia Richards Suelzer, Wolters Kluwer, a global information services company:  “We can now make catastrophic miscalculations in nanoseconds and broadcast them universally. We have lost the balance inherent in ‘lag time.’”

The CIO of Wal-Mart is quoted upfront in the report (from The Economist in 2010!)—“Every day I wake up and ask, ‘how can I flow data better, manage data better, analyze data better?’”—but only 10% of survey respondents identified themselves as “technology developers or administrators.” A plurality of the survey participants (40%) identified themselves as a research scientist or as employed by a college or university.

Which means that very few IT executives—the people who actually manage the data—or very few business intelligence and analytics professionals or data scientists—the people who actually find out what the data means— were asked to respond to the survey’s big data scenarios.  Apparently they don’t qualify as “Internet experts.”

That’s a pity because the practitioner’s view could help, possibly more than the “Internet expert” view,  maintain a more balanced and nuanced view of the good and bad of big data, and certainly tone down the made-for-the-masses “ugly scenario,” in which big data is equated with big brother.  Without the experienced-based opinions of the people who live and love data, who know well its potential and limitations, the Pew/Elon study advances our understanding of the future of big data as much as today’s (July 29) Dilbert cartoon, in which the pointy-haired boss  says ”Big data lives in the cloud. It knows what we do… If we accept big data in our serves, we will be saved from bankruptcy.”

So where do you see big data in 2020? Will it be useful, harmful, or neutral for our well-being? Is it going to help organizations in making better decisions or will it prove to be a low-return or no-return investment? Is it a promising “new science” or simply a steady evolution in our ability to analyze data?