GSA USASearch Wins 2011 Government Big Data Solutions Award

The Government Big Data Solutions Award was established to highlight innovative solutions and facilitate the exchange of best practices, lessons learned and creative ideas for addressing Big Data challenges. The top five nominees and overall winner was announced at Hadoop World in New York City on November 8 2011.

The Government Big Data Solutions Award Program is coordinated by CTOLabs.com.  The 2011 judging panel included: Doug Cutting, creator of Apache Hadoop and architect at Cloudera, Alan Wade, former CIA and IC CIO, Ryan LaSalle, Director of Accenture Cyber R & D, Ed Granstedt, Senior VP Director of the QinetiQ Strategic Solutions Center and Chris Dorobek, founder, editor and publisher of DorobekInsider.com.

The top five honorees of the Government Big Data Solutions Award are:

• USA Search: Hosted search services over more than 500 government sites.  Provides search and suggestion services plus analytical tool dashboards.
• GCE Federal: Cloud-based financial management solutions.
• PNNL Bioinformatics: Advancing understanding of health, biology, genetics and computing.
• SherpaSurfing: A cybersecurity solution that analyzes trends, finds malware, and writes alerts.
• US Department of State, Bureau of Counselor Affairs: Large data set with critically important applications for citizen service and national security.

The winner of the 2011 Government Big Data Solutions Award is the USA Search Program of the US General Services Administration Office of Citizen Services and Innovative Technologies.

Award judges saw the USASearch Program as a great example of solving Big Data problems to improve government agility and provide better service for less. In line with the GSA’s cost-saving “build once, use many times” paradigm, USASearch has provided hosted search services for USA.gov and, through its Affiliate Program, over 500 other government websites. This is done in an incredibly cost-effective way, especially for the agencies involved (which receive these services from GSA for free).

In 2010, USASearch adopted an open architectural model to better exploit shared solutions and open source technology. This model leverages Cloudera’s Distribution including Apache Hadoop (CDH3). The move brought further cost savings and scalable shared search services, which drove up usage. USASearch adopted Hadoop to improve the search results by aggregating and analyzing  information on what users are searching, their success in finding it, the time of the search, the affiliate, the results, and which results users click on, among other trends. To do so across hundreds of affiliates with growing traffic, USASearch considered scaling up or dividing its database systems, but they knew that these solutions would be costly and temporary.  Instead, it turned to  HDFS, Hadoop, and Apache Hive—a big data system that could grow cost effectively and without downtime, be naturally resilient to failures, and sensibly handle backups.

The overhaul of USASearch’s analytics is a dramatic success story. In the space of a few months, USASearch went from having a brittle and hard-to-scale RDBMS-based analytics platform to an agile, scalable Hadoop-based system.  By using a state-of-the-art open source technology, USASearch has created a radically different search service that transforms the customer experience Hadoop’s uses and effects will continue to expand as more data sources and tools are added. Having a government-owned and controlled search service allows provides insight on the needs and concerns of Americans to drive enhancements to other delivery channels. The public has a much improved experience when interacting with the government due to USASearch.

The GSA is to be congratulated for their mission-focused, citizen-centered, open approach to a big data challenge and a resulting solution that improves the experience of a broad swath of users of federal services. On behalf of our judges and the many citizens who use this capability on a daily basis we say thank you, and congratulations on this well deserved recognition.