Assisted Insight: The Future of Data Discovery

Businesses don’t want “data discovery”, “visualization” or “analytics” tools – businesses want one thing: insight.

The challenge has been how to provide that insight. Twenty years ago the processing power of machines was limited. To overcome this limitation, traditional business intelligence providers created cubes and OLAP.

Businesses don’t want “data discovery”, “visualization” or “analytics” tools – businesses want one thing: insight.

The challenge has been how to provide that insight. Twenty years ago the processing power of machines was limited. To overcome this limitation, traditional business intelligence providers created cubes and OLAP.

Over the past decade, those limitations have lifted and interactive data visualizations tools that use in-memory data models have blossomed. The paradigm shifted, and with it, new powerhouses like Tableau Software have arisen.

In the next decade an equally profound shift will occur. As processing power increases and cloud computing takes hold, new capabilities will emerge—new capabilities that will drive new paradigms. One of those will be Assisted Insight.

What is Assisted Insight?

Assisted insight is the process of presenting users with computer-generated insights in a way that guides them when analyzing data. Using a combination of framing, algorithms and visualizations, assisted insight will accelerate the discovery of new insights and uncover insights that previously remained hidden.

With new user interfaces, users will frame their problem in business terms instead of mathematical terms. This will allow algorithms to dig deeper and visualizations to be presented to maximize new insights. If users enter their assumptions and expectations, data discovery software can use that framing to find the unexpected.

By combining this framing with statistical and machine learning algorithms, data discovery software will create a list of potential insights, ranked by confidence, for a user to explore. With the massive amount of computing power users have at their disposal today, thousands of potential data combinations can already be explored by algorithms in the same time it would take a user to explore one manually.

Once insights have been identified, the framing and nature of the insight will be used to configure interactive visualizations that communicate those insights to the user using best practices. Users will explore these visualizations to verify the insight and gain a nuanced understanding of it.

Finally, users will use the visualizations to refine the insights by changing the parameters of the analysis and dig deeper into the data. A cycle emerges of the user refining their framing of the problem, software proposing potential insights, and users exploring & refining those insights further.

Framing the Problem

Framing refers to how a user defines their analysis. Traditionally, data analysis is done using a “mathematical” frame. Metrics are low or high, statistics like averages and medians are calculated (assuming distributions are bell-shaped) and the user is presented with a series of numbers that they need to translate into their problem space.

By framing the analysis using a “problem” frame, a user can work directly in the concepts they use to drive their business: metrics are “good” or “bad”, distributions are “normal” or “abnormal”, values are “expected” or “unexpected”, and the user is presented with visualizations configured to communicate the conceptual space of the problem.

Generating Insights with Algorithms

Algorithms can help users quickly explore potential combinations and correlations within their data to find unknown trends, anomalies and relationships. But algorithms are only as good as their input—garbage in, garbage out.

By adding the meaning of the data—the assumptions, expectations and known relationships—into the analysis, algorithms can help users find business insights, not mathematical insights.

Using Visualization to Refine Insights

Visualization helps make data consumable. By using colors, shapes and other visual aspects that align with the meaning of the data, comprehension of insights is enhanced. Users can be shown insights faster by highlighting potential insights and being given natural language explanations.

But not all potential insights will be valuable. Most won’t be.

Visualizations can give the context of an insight to a user so they gain a nuanced understanding and a spring pad for further exploration. Through direct manipulation and other techniques, users can refine not only the visualization itself, but the framing of the problem. This new framing can be used to drive the algorithms to create new potential insights.

Assisted Insight Today

At present, no one has combined all these elements to create true assisted insight. But companies are moving fast to develop these capabilities.

IT operations analytics vendors like Netuitive have developed behavior learning engines that use algorithms to learn what is “normal” vs “abnormal” in real-time, machine-generated data. Using a constrained problem space allows them to build the framing of the problem directly into their software. Making this generic requires new user interfaces to allow users to enter their framing, and new algorithms for data problems that don’t map easily onto the IT operations analytics space.

Business intelligence startups have started developing “automated insight” functionality by using algorithms to find potential insights and automatically configure visualizations for those insights.

Within data discovery, vendors have focused on natural language queries and storyboarding—but these don’t help uncover new insights. Natural language queries speed the process of configuring a visualization once a user has an idea what to look for, but do nothing to help users find hidden insights. Story-boarding helps to communicate insights once they are found, not to generate new ones.

The Future of Assisted Insight

It will take several more years before assisted insight becomes a standard feature of business intelligence and data discovery suites. The vendors who develop it early will have the edge. They’ll have had more time to improve their user interfaces, refine their algorithms and develop new visualizations.

But the pieces are there, just waiting for someone to put them together. Over the next few months, I’ll be writing about those pieces and how to combine them. Follow me to get notified about future posts.