BI’s Dirty Secrets

The overwhelming amount of articles, seminars, white papers and product announcements on big data, predictive analytics, cloud and mobile business intelligence might make you think that every business person is walking around with real-time dashboards on their tablets,

The overwhelming amount of articles, seminars, white papers and product announcements on big data, predictive analytics, cloud and mobile business intelligence might make you think that every business person is walking around with real-time dashboards on their tablets,  figuring out how to get more revenue, improve customer satisfaction and increase profitability.

IDC recently stated “In 2011 the business analytics technology market, which includes software, hardware, and services for data warehousing, query, reporting, analysis, advanced analytics, content analysis, analytics spatial information management, and analytic applications, surpassed $90 billion in worldwide revenue.” Business intelligence (BI) has ranked in the top tier of IT application priorities for over a decade.  Vendors have made significant improvements to BI, data integration and data warehousing products during that time. Experts have established best practices in these areas.

Yet despite the attention, technological capabilities and investments there are dirty secrets in our industry that analysts and pundits do not like to admit:

• Spreadsheets are the BI tool of choice for business people. Many enterprises use several BI tools, yet the only pervasive BI tool is a spreadsheet.
• Manually coded extracts are the predominant data integration tool to get data into those spreadsheets (and often into the databases) used for reporting. IT often used a data integration tool to get data into a data warehouse, but to then get data into data marts, OLAP cubes or other databases or tables used for reporting IT uses SQL scripting. Business people use the capability that Microsoft Excel offers to gather data and often supplement this by leveraging Microsoft Access.

Many are in denial about these dirty secrets, but in order to change something one must first recognize there are problems. When IT and analysts do confront these dirty secrets they often lash out at spreadsheets and SQL, but the reality is the use of these are symptoms of problems, not the problems themselves.

The result of this situation is that managing the information that business people need takes longer and costs more. In addition, there is a perpetual and increasing backlog to obtain comprehensive, consistent and current business information.

Subsequent posts will discuss the causes of these dirty secrets and what both IT and business need to do differently.