Bigger Data, Better Intelligence for Government

Tomorrow (at 2PM Eastern on Thursday), the White House White House Office of Science and Technology Policy will host a 90-minute forum on Challenges and Opportunities in Big Data.

Tomorrow (at 2PM Eastern on Thursday), the White House White House Office of Science and Technology Policy will host a 90-minute forum on Challenges and Opportunities in Big Data. You can watch the event live at the previous link, and see federal government science heads from OSTP, NSF, NIH, DoE, DoD, DARPA and USGS outline how their agencies are engaged in Big Data research. The event will also include a discussion by a panel of thought leaders from academia and industry, moderated by Steve Lohr of the New York Times.

Naturally, the event has prompted some speculation about the government’s “big data agenda“, but this in fact a direction the administration has been pushing ever since the Obama appointed Aneesh Chopra as the nation’s first CTO in 2009. (Chopra passed the mantle to current CTO Todd Park just this this month.) One of the first initatives was the creation of data.gov, a vehicle to make government data available in a timely and accessible format for a community of citizen data scientists to make sense of it all. For example, independent analysis of data from data.gov (using Big Data statistical analysis techniques) revealed a flaw in the 2000 Census results that went unnoticed by the Census Bureau.

And there’s the rub: while much of the talk around Big Data has been around storage and search for massive data sets, the real value comes from analysis and inference on the data themselves. I’d expect to hear more from the administration on this topic in today’s forum. The administration is already pushing to use more open-source software in government, the open-source R language (along with the big-data enhanced distribution from Revolution Analytics) is already widely used at NIST, FDA, NOAA, CIA, and many other government agencies. Given that the data storage and data analysis infrastructure is already well in place, I expect we’ll hear a vision for new kinds of applications, citizen services and data-driven strategies from the participating federal agencies in today’s forum.