Can You Predict Crowd Behavior? Big Data Can

The 2012 elections were a hotbed of predictions, but one man’s election forecast stood out from the rest both in its boldness and in how wildly accurate it turned out to be. Nate Silver, a blogger and statistician gave then presidential incumbent, Barack Obama, more than a 90 percent chance of winning the election. In addition, in his last prediction he gave Mr. Obama 313 votes and Mr.

The 2012 elections were a hotbed of predictions, but one man’s election forecast stood out from the rest both in its boldness and in how wildly accurate it turned out to be. Nate Silver, a blogger and statistician gave then presidential incumbent, Barack Obama, more than a 90 percent chance of winning the election. In addition, in his last prediction he gave Mr. Obama 313 votes and Mr. Romney 225 votes with a margin of 14. The actual election results fell directly within this prediction with the incumbent garnering 303 electoral votes and Mitt Romney winning 206. While many disparaged Silver’s predictions as a ploy to try to swing the election (Silver was an open Obama supporter), it seemed Silver got the last laugh when his prediction came true.

Crunching huge data sets to predict crowd behavior may still seem like a phenomena of the future or to some, impossible, but the reality is that future is already upon us and has been for several years now. We’ve all heard about big data and read the articles on how Hadoop allows organizations to capture and harness big data in an affordable manner, but we need to start thinking bigger of what the true implications are of these advancements. As big data services move to the cloud, making it possible for even the smallest business to have access to big data insights, operating based on predictions could become more commonplace than we realize.

The Future Came Several Years Ago

Back in January 2010 a small startup company called Recorded Future released a blog post claiming that Yemen would likely have food shortages and flooding that year. Due to the combination, the country was headed for conflict. By September of that year not only had Yemen experienced flooding but was also combating food shortages, which led to protestors in the streets calling for the resignation of the country’s president. By February 2012, the protests had turned violent with protesters killed by gunmen and the Yemen President suffering severe injuries after a bomb was planted in his compound.

How did Recorded Future predict this uprising? Were they foreign policy experts familiar with the area? No, they were data scientists that rely on publicly available information such as news articles and tweets on Twitter to make their predictions.

Could it have simply been an educated, yet lucky, guess? Possibly, but so far the organization has predicted large protests in 19 countries in Asia, Africa and Europe with high accuracy. They also have a relationship with the CIA, banks and hedge funds making the intersection of big data and predictive analytics more of a reality than hype.

The Implications of Predicting Crowd Behavior

The implications of predictive analysis go beyond large government agencies and financial institutions. Businesses, small and large, can embrace predictive analytics as well. Marketing, in particular, has already started to explore this realm by using consumer data to predict which customers are experiencing a particular life event, such as Target’s much publicized targeted ads to expecting women. More and more businesses are able to segment their consumers not just based on demographic information but on their individual interests, and yes, even future events in their lives.

This revolution isn’t limited to marketing, however. Cell phone companies can use big data to determine when there will be a spike in cell phone use due to triggering events, and manufacturers can better evaluate when a piece of machinery needs maintenance without having to physically inspect the machine.

Truly, there isn’t a sector from health care to Wall Street to government that couldn’t benefit, but aren’t there some negative implications from this? Aren’t there privacy concerns to be addressed? WIll organizations not only be predicting the future but even changing the future based on their efforts to sway public sentiment based off what the data says?

Privacy is certainly an issue, but the consumer seems perfectly willing to give up privacy, at least in some areas, in order to be catered to and quite frankly politicians and businesses have been trying to sway the future forever. After all, is running an ad for a product in order to get the consumer, who otherwise would be completely unaware it existed, to buy it really any different from running a highly individualized ad to a woman who is expecting a baby? Both seek to modify behavior. The real difference lies in how effective the two methods are, and don’t forget, despite President Obama’s own efforts to use big data to target voters, a huge percentage of the population still voted for Romney reminding us the individual still has the true deciding power.

Despite the naysayers and conspiracy theorists, predictive analytics has a huge potential for good, and we are only just seeing the beginning of what we can do with big data.