The Promise and Challenge of Using Big Data to Address World Problems

Author: Linda Rosencrance
Spotfire Blogging Team

Last month, we wrote that as data and analytic tools become more prevalent in society, the consumerization of data has opened up a whirlwind of opportunities for quality of life improvements around the world. In this same spirit, we’re blogging about additional options for people across the globe to use data and tools to learn about and contribute to a variety of quality of life improvements.

“Done well, big data has the potential to advance important social goals in areas such as disease surveillance, student curricula, and microcredit. Understanding big data’s potential for social impact and the barriers to capturing it are the first steps toward its effective use.”

That’s the last paragraph and the crux of this article about harnessing big data to address the world’s problems.

For instance, Engineering Social Systems’ project in Nairobi has modeled the growth of slums and informal settlements using geo-coded mobile-phone transaction data. The article points out that the city government could use this type of data to better allocate municipal resources as well as the limited funding set aside for infrastructure.

As the examples in the article suggest, the social organizations that integrate big data collected by private industry, government and others into their strategies can make significant differences in the lives of people worldwide.

However, to do that, these organizations have to overcome a few barriers including getting access to the data as well as identifying or developing the right datasets, not to mention finding people with the critical analytical skills needed to analyze the data—people who are in short supply at the moment.

A lot of this important data is held by governments that might not even fully use it themselves or by private companies that don’t want to release it for proprietary reasons. The “open data” movement at organizations like the World Bank is a good start, but more needs to be done to give these organizations access to this information.

Then there’s the problem of data compatibility and quality. It’s not easy to integrate data stored on legacy systems and other disparate systems. Not only that, but the quality of much of the data just isn’t very good. Those issues aside, a number of the interested social organizations don’t even have the hardware and software to “integrate, analyze and visualize big data.” Cloud computing, though, could tackle some of these problems.

And, as we’ve mentioned in the past, there’s a major shortage of analytical talent—a shortage that presents these social organizations with an additional hurdle. According to the McKinsey Global Institute (MGI), this is already an issue in the private sector. MGI predicts that in the US alone there will be an annual shortage of graduates in deep analytical fields of 140,000 to 190,000 by 2018.

And that’s not good news for organizations looking to make the world a better place, according to the article. Despite the efforts of organizations such as Data Without Borders, which matches non-profits in need of data analysis with freelance and pro bono data scientists, dealing with the shortage of data analysts may be even more challenging in the social sector.

In our next post, we’ll continue to discuss how data advancements can improve the quality of our lives. Subscribe to our blog to be alerted.