Shedding Light on Dark Data: How to Get Started

Move over Big Data. There’s a new buzzword: dark data.

It’s actually not so new—Gartner coined the term a couple years back—but dark data is finally starting to catch on in market research circles and it represents a huge untapped opportunity for insights!

Gartner defined dark data as “the information assets organizations collect, process, and store during regular business activities, but generally, fail to use for other purposes.”

Move over Big Data. There’s a new buzzword: dark data.

It’s actually not so new—Gartner coined the term a couple years back—but dark data is finally starting to catch on in market research circles and it represents a huge untapped opportunity for insights!

Gartner defined dark data as “the information assets organizations collect, process, and store during regular business activities, but generally, fail to use for other purposes.”

The definition has since expanded to encompass not just internal data, but the broader spectrum of data that are readily available to organizations.

The common denominators are 1) these data are largely unstructured and 2) they are not being analyzed. In fact, according to IDC, 90% of the unstructured data are never analyzed!

Why Search in the Dark?

Maybe you’ve heard this one?

A police officer comes upon a man crawling around on all fours under a streetlight one evening.

The man explains that he’s looking for his wallet.

“Where do you think you lost it?” asks the policeman.

“Across the street, but the light is so much better here,” says the man.

Popular among data scientists, I think this joke illustrates the irrationality of a lot of common thinking in research these days. We tend to search for insights in a relatively limited but easily accessible location—survey data—as if the only answers to be found must be there.

And even that relatively small pond isn’t being thoroughly fished. As I’ve blogged in the past, for most of us, even survey open-ends/comment data are still “dark data”!

At the risk of deluging you with metaphors, the fact remains that what we can find in our survey data is only the tip of the insights iceberg.

We have at our disposal all manner of unstructured data for which text analytics are uniquely suited to organize and understand, including images and video—without any enrichment or visual content analysis. For example, images often contain file name and metadata descriptions in text format that can be analyzed with software like OdinText. Videos, too, often contain transcript data, and there are technologies like YouTube’s, which can handle audio-to-text translation.

A Few Things to Consider

Dark data can be Big Data. And very Big Dark Data can prove daunting (that’s partly why it stays dark in the first place).

But dark data can also be quite small we’ve found.

And just as Big Data isn’t necessarily valuable just because it’s big, dark data certainly isn’t valuable just because it’s dark.

Lastly, technology can’t make garbage data valuable and the complexities involved in analyzing some forms of dark data often require taking a sample or deciding exactly which parts of the data might prove most interesting to analyze.

Don’t Be Afraid of the Dark

There are tons of ways to start putting dark data to work for your organization. Here are recent examples of how clients are using OdinText currently to shed light on their dark data.

Communications

Phone transcripts, chat logs, and email are often dark data that text analytics can help illuminate. Would it be helpful to understand how personnel deal with incoming customer questions? Which of your products are discussed with which of your other products or competitors’ products more often? What problems or opportunities are mentioned in conjunction with them? Are there any patterns over time?

We already have clients doing these types of analyses with OdinText. It is almost always exploratory at first, but these clients recognize the need to look.

Merging Disparate Dark Data Sources

How about integrating, say, audio file transcripts from a call center with click data from websites? There are plenty of cases where merging dark data sources can yield important insights that would not be attainable using conventional tools.

In such a case, you would typically start with the goal of understanding one or more KPIs. Thinking about what data you might have available to help understand, model and predict these would be the next step. How similar are these data, again, what is the value to understanding said KPI/s?

Ideally, the data that is joined is similar in some respects, but it doesn’t necessarily have to be perfect.  We may be willing to overlook various problems in this data in hopes that the aggregate data (which may involve dropping in means, merging various text fields in different ways, etc.) will give us a better understanding of how to affect and manage against our KPI/s.

Again, I must stress that even this does not necessarily need to involve/yield Big Data. For instance, if you are a pharmaceutical company and the data in question are drug tests or small samples of doctors, even after the merge the data will still be relatively small by most standards.

Also, the data need not be any more sophisticated than simple survey data or even in-depth interviews over the span of, say, 2-3 years. That said, it is always more interesting if marketing research opinion data—whether survey or some sort of more qualitative data—is accompanied by some real behavior or outcome like efficacy or sales.

My opinion on this sort of analysis has recently changed drastically as our clients have shown us that where there is a will, there is often both a way and one or more very lucrative insights!