From “The Farm” to FarmVille - SmartData Collective

In an experiment on crowdsourcing intelligence, Applied Research Associates, Inc hopes to see if it can build better intelligence collectors and analysts than the CIA produces on “The Farm”, its field academy for clandestine officers, with a tool that bears more resemblance to Facebook’s popular social networking game FarmVille. ARA recently launched the Aggregative Contingent Estimation System (ACES), a website funded by the Intelligence Advanced Research Projects Activity (IARPA) where users can vote on future social, political, scientific, economic, and military developments similar to an online poll, with the results then aggregated into predictions and probabilities. IARPA hopes to use ACES to test the value of crowdsourcing for national intelligence.

The idea that the sum of novice opinions may be more powerful than that of an expert or that all of us know more than any of us is not new. Markets, which represent an aggregate of public opinion, have been shown to be a good predictor of the future. Perhaps the most striking example came after the Space Shuttle Challenger Disaster where a failure of the O-ring seals in its booster rockets caused the shuttle to explode on takeoff. Almost instantly, the stocks of the four companies that had built the shuttle plummeted, but by the end of the day, all but one had recovered, though no information on the catastrophe had been announced. Six months later, when the investigation into the disaster was complete and the cause of the explosion was released, it was revealed that the company which the market had singled out was responsible for the failure. Nobody knows how the market knew before the report – perhaps some insider information had been leaked and made its way around, perhaps a number of people had a hunch – but it had accurately and independently predicted the findings.

Numerous other programs have already sought to harness collective intelligence for national security. The Total Information Awareness project, for example, involved Futures Markets Applied to Prediction (FutureMAP), a “market in the future of the Middle East” where users could trade possible political developments to earn real money. FutureMAP was shut down due to congressional outrage over a program that essentially allowed people to bet on terrorist attacks and assassinations. Experiments with the market approach and crowdsourcing continued, however, with efforts such as In-Q-Tel‘s current experiment with a system where participants can bet on cyber security issues to predict the future of cryptography research, threats, and regulations.

ACES, however, uses a unique approach similar to social gaming to fine-tune predictions. No money changes hands on ACES’ website, only bragging rights, as users compete to produce the most accurate predictions. Over time, the website shows them where they stand, for example that events they predict to be 60% likely only occur 40% of the time, letting them adjust their confidence down. This approach has numerous advantages, as it teaches the online analysts to get better while determining which users are best. The predictions of users that have been more accurate in the past are weighted more than those of less succesful participants, making the aggregates more accurate and encouraging users to compete with their friends. This competive element makes it similar to hugely popular social games such as FarmVille, which could help the website attract and engage participants.

If ACES can attract a following as large, serious, or devoted as the Facebook farming simulator, it just might work.