Big Data Analytics, Business Intelligence and the Mind of Sherlock Holmes

“My name is Sherlock Holmes. It is my business to know what other people do not know.”
— The Adventure of the Blue Carbuncle 

“My name is Sherlock Holmes. It is my business to know what other people do not know.”
— The Adventure of the Blue Carbuncle 

Sherlock Holmes turned 125 years old last year, and he’s never been more alive and well. The world seems more captivated by Sir Arthur Conan Doyle’s legendary London detective than ever before. Much of this excitement has been driven recently by the smash BBC One TV series Sherlock, drawing rave reviews for its update of Holmes and Dr. Watson as present-day Londoners fighting 21st-century crime. (Similarly, the U.S. version of the series, Elementary, is also a major new hit.)

Pop culture critic and author John Powers cleverly explains Holmes’ enduring appeal as a literary hero and cultural icon:

Sherlock Holmes “possesses no superpowers — his parents weren’t wizards, no radioactive spider bit him — [and yet] his gifts are cool enough to be superhuman. Playing to our fantasies of being smarter than everyone else, Holmes performs jaw-dropping feats of perception.”

It’s no coincidence that heightened interest in Sherlock Holmes coincides with the rapidly accelerating, proliferating sources of information around us: databases, documents/text, big data, social media, web content and more. Like Sherlock Holmes, we all want to make sense of seemingly unrelated information and “be smarter than everyone else” — or at least outsmart the competition, outsmart criminals and fraudsters, outsmart seemingly intractable business problems.

A quick review of Conan Doyle’s novels and short stories reveals Sherlock Holmes shared useful advice on effectively accessing, analyzing — and yes, unifying — information. His advice rings truer than ever in today’s increasingly information-rich but insight-deficient world.

Sherlock Holmes on Big Data Analytics and Information Management

Now the skillful workman is very careful as to what he takes into his brain-attic. He will have nothing but the tools which may help him in doing his work, but of these he has a large assortment, and all in the most perfect order. — A Study in Scarlet

Holmes draws a wise distinction regarding information of direct, immediate impact that one should remain continuously aware of and be ready to act upon. And today there is indeed “a large assortment” of information that exists across a wide assortment of sources — databases, CMS, email, SharePoint, web and other information silos:

A man should keep his little brain-attic stocked with all the furniture that he is likely to use, and the rest he can put away in the lumber-room of his library, where he can get it if he wants it. — The Five Orange Pips

A “lumber room” in Holmes’ late 19th century Britain stored replaced furniture and related items, particularly in a wealthy Briton’s mansion. As all furniture was custom-made and of possible future use, it would be stored rather than sold or discarded. With the advent of innovations including Hadoop, organizations now have Big Data “lumber rooms” that enable efficient, cost-effective capture and retention of huge volumes of information.

Bringing “perfect order” to these far-flung, siloed information sources by readily combining them for easy access and analysis remains one of today’s most critical challenges. It is also the very mission of unified information access technology, led by Attivio’s Active Intelligence Engine, which freely integrates, correlates and presents data and content alike, leaving no vital information behind.

Sherlock Holmes on Analytic Thinking and Agile Business Intelligence

It is of the highest importance… to be able to recognize, out of a number of facts, which are incidental and which vital. Otherwise your energy and attention must be dissipated instead of being concentrated. — The Reigate Puzzle

For decades, business intelligence (BI) systems have provided managers with reports and dashboards that boil down detailed structured data (databases, data warehouses) into performance metrics trended over time — in an effort to provide quick focus on the vital facts.

However, KPIs alone cannot tell you the whole story about the business; even worse, misguided managers may end up superficially ‘managing to the metric’ instead of managing the business itself:

You see, but you do not observe. The distinction is clear. — A Scandal in Bohemia

As an example I explored in a recent article, Starbucks CEO Howard Schultz wrote in 2008 that Starbucks’ had lost its way in large part due to management overlooking ongoing business missteps in favor of focusing on a single metric which proved to be a poor indicator of the company’s true health:

There is nothing more deceptive than an obvious fact. — The Boscombe Valley Mystery

Simply put, the numbers can tell you what is happening, but the most effective managers of leading organizations will also insist on understanding why.

There are few people able to deduce what the steps were which led up to a given result. This is the power of reasoning backwards, or analytically. — A Study in Scarlet (paraphrased)

The most successful managers are those who think analytically; they refuse to merely accept performance metrics at face value, choosing instead to gain a deep, “root-level” understanding of the company’s operations and customers. Doing so requires asking probing, in-depth “get your hands dirty” business questions. Getting the answers to such vital questions requires the ability to go beyond numbers alone and gain complete agile business intelligence drawn from the entire spectrum of enterprise information — structured and unstructured, internal and external.

Attivio’s Active Intelligence Engine empowers managers with the ability to “analyze everything” by integrating and correlating all silos of structured data and unstructured content, and exposing this unified information to popular BI tools, through native SQL support and ODBC/JDBC connectivity, for powerful self-service visualization and analytics.

On a final related note, one of the most memorable Sherlock Holmes stories featured the detective solving the case of a stolen racehorse and its murdered trainer:

[Police inspector:] “Is there any point to which you would wish to draw my attention?”
[Sherlock Holmes:] “To the curious incident of the dog in the night-time.”
“The dog did nothing in the night-time.”
“That was the curious incident.” — Silver Blaze

Holmes solved the mystery in part by observing the guard dog did not bark, concluding the intruder was not a stranger to the dog. Sherlock Holmes’ brilliance lies in his uncanny ability to carefully observe information and join together seemingly unrelated facts to assemble a complete picture of a crime.

Attivio empowers today’s managers and executives with a similar ability to work with a complete business picture, by JOINing and presenting all related data and content matching a user query — on an ad hoc basis, with no advance data modeling required. By unifying and presenting all enterprise data and content, Attivio provides a complete business picture that enables Big Data analytics and insights to solve new challenges:

If you have all the details of a thousand [past crimes] at your finger ends, it is odd if you can’t unravel the thousand and first. — A Study in Scarlet

Learn more: Read the white paper “Analyze Everything”: Driving Agile and Complete BI.