Cookies help us display personalized product recommendations and ensure you have great shopping experience.

By using this site, you agree to the Privacy Policy and Terms of Use.
Accept
SmartData CollectiveSmartData Collective
  • Analytics
    AnalyticsShow More
    sales and data analytics
    How Data Analytics Improves Lead Management and Sales Results
    9 Min Read
    data analytics and truck accident claims
    How Data Analytics Reduces Truck Accidents and Speeds Up Claims
    7 Min Read
    predictive analytics for interior designers
    Interior Designers Boost Profits with Predictive Analytics
    8 Min Read
    image fx (67)
    Improving LinkedIn Ad Strategies with Data Analytics
    9 Min Read
    big data and remote work
    Data Helps Speech-Language Pathologists Deliver Better Results
    6 Min Read
  • Big Data
  • BI
  • Exclusive
  • IT
  • Marketing
  • Software
Search
© 2008-25 SmartData Collective. All Rights Reserved.
Reading: Shedding Light on Dark Data: How to Get Started
Share
Notification
Font ResizerAa
SmartData CollectiveSmartData Collective
Font ResizerAa
Search
  • About
  • Help
  • Privacy
Follow US
© 2008-23 SmartData Collective. All Rights Reserved.
SmartData Collective > Big Data > Data Mining > Shedding Light on Dark Data: How to Get Started
Big DataData MiningData Quality

Shedding Light on Dark Data: How to Get Started

TomAnderson
TomAnderson
7 Min Read
SHARE

Move over Big Data. There’s a new buzzword: dark data.

It’s actually not so new—Gartner coined the term a couple years back—but dark data is finally starting to catch on in market research circles and it represents a huge untapped opportunity for insights!

Gartner defined dark data as “the information assets organizations collect, process, and store during regular business activities, but generally, fail to use for other purposes.”

More Read

data-driven startups
A Proven Template For Financing Data-Driven Startups
Defending Your Analytics: Handling Hecklers
What Cyber Criminals Can Do With Your Metadata
Seven Steps to Heaven: Marketing Automation with a CRM System [INFOGRAPHIC]
Is Artificial Intelligence About to Change Doing Business Forever?

Move over Big Data. There’s a new buzzword: dark data.

It’s actually not so new—Gartner coined the term a couple years back—but dark data is finally starting to catch on in market research circles and it represents a huge untapped opportunity for insights!

Gartner defined dark data as “the information assets organizations collect, process, and store during regular business activities, but generally, fail to use for other purposes.”

The definition has since expanded to encompass not just internal data, but the broader spectrum of data that are readily available to organizations.

dark-data-text-analytics

The common denominators are 1) these data are largely unstructured and 2) they are not being analyzed. In fact, according to IDC, 90% of the unstructured data are never analyzed!

Why Search in the Dark?

dark-data-lamp-postMaybe you’ve heard this one?

A police officer comes upon a man crawling around on all fours under a streetlight one evening.

The man explains that he’s looking for his wallet.

“Where do you think you lost it?” asks the policeman.

“Across the street, but the light is so much better here,” says the man.

Popular among data scientists, I think this joke illustrates the irrationality of a lot of common thinking in research these days. We tend to search for insights in a relatively limited but easily accessible location—survey data—as if the only answers to be found must be there.

And even that relatively small pond isn’t being thoroughly fished. As I’ve blogged in the past, for most of us, even survey open-ends/comment data are still “dark data”!

At the risk of deluging you with metaphors, the fact remains that what we can find in our survey data is only the tip of the insights iceberg.

dark-data-text-aanalytics-ice-berg

We have at our disposal all manner of unstructured data for which text analytics are uniquely suited to organize and understand, including images and video—without any enrichment or visual content analysis. For example, images often contain file name and metadata descriptions in text format that can be analyzed with software like OdinText. Videos, too, often contain transcript data, and there are technologies like YouTube’s, which can handle audio-to-text translation.

A Few Things to Consider

Dark data can be Big Data. And very Big Dark Data can prove daunting (that’s partly why it stays dark in the first place).

But dark data can also be quite small we’ve found.

And just as Big Data isn’t necessarily valuable just because it’s big, dark data certainly isn’t valuable just because it’s dark.

Lastly, technology can’t make garbage data valuable and the complexities involved in analyzing some forms of dark data often require taking a sample or deciding exactly which parts of the data might prove most interesting to analyze.

Don’t Be Afraid of the Dark

There are tons of ways to start putting dark data to work for your organization. Here are recent examples of how clients are using OdinText currently to shed light on their dark data.

Communications

Phone transcripts, chat logs, and email are often dark data that text analytics can help illuminate. Would it be helpful to understand how personnel deal with incoming customer questions? Which of your products are discussed with which of your other products or competitors’ products more often? What problems or opportunities are mentioned in conjunction with them? Are there any patterns over time?

We already have clients doing these types of analyses with OdinText. It is almost always exploratory at first, but these clients recognize the need to look.

Merging Disparate Dark Data Sources

How about integrating, say, audio file transcripts from a call center with click data from websites? There are plenty of cases where merging dark data sources can yield important insights that would not be attainable using conventional tools.

In such a case, you would typically start with the goal of understanding one or more KPIs. Thinking about what data you might have available to help understand, model and predict these would be the next step. How similar are these data, again, what is the value to understanding said KPI/s?

Ideally, the data that is joined is similar in some respects, but it doesn’t necessarily have to be perfect.  We may be willing to overlook various problems in this data in hopes that the aggregate data (which may involve dropping in means, merging various text fields in different ways, etc.) will give us a better understanding of how to affect and manage against our KPI/s.

Again, I must stress that even this does not necessarily need to involve/yield Big Data. For instance, if you are a pharmaceutical company and the data in question are drug tests or small samples of doctors, even after the merge the data will still be relatively small by most standards.

Also, the data need not be any more sophisticated than simple survey data or even in-depth interviews over the span of, say, 2-3 years. That said, it is always more interesting if marketing research opinion data—whether survey or some sort of more qualitative data—is accompanied by some real behavior or outcome like efficacy or sales.

My opinion on this sort of analysis has recently changed drastically as our clients have shown us that where there is a will, there is often both a way and one or more very lucrative insights!

Share This Article
Facebook Pinterest LinkedIn
Share

Follow us on Facebook

Latest News

sales and data analytics
How Data Analytics Improves Lead Management and Sales Results
Analytics Big Data Exclusive
ai in marketing
How AI and Smart Platforms Improve Email Marketing
Artificial Intelligence Exclusive Marketing
AI Document Verification for Legal Firms: Importance & Top Tools
AI Document Verification for Legal Firms: Importance & Top Tools
Artificial Intelligence Exclusive
AI supply chain
AI Tools Are Strengthening Global Supply Chains
Artificial Intelligence Exclusive

Stay Connected

1.2kFollowersLike
33.7kFollowersFollow
222FollowersPin

You Might also Like

business systems for data driven businesses
Big Data

Business Management Systems for Data-Driven Businesses

9 Min Read
big data in seo ecommerce
Big DataExclusive

5 Ways Big Data Fuels SEO For eCommerce Stores

7 Min Read

How Business Analytics Can Lead to That ‘Aha’ Moment

4 Min Read

O’Reilly Chums the Water: Ken Hilburn Rises to the Bait

3 Min Read

SmartData Collective is one of the largest & trusted community covering technical content about Big Data, BI, Cloud, Analytics, Artificial Intelligence, IoT & more.

ai is improving the safety of cars
From Bolts to Bots: How AI Is Fortifying the Automotive Industry
Artificial Intelligence
ai in ecommerce
Artificial Intelligence for eCommerce: A Closer Look
Artificial Intelligence

Quick Link

  • About
  • Contact
  • Privacy
Follow US
© 2008-25 SmartData Collective. All Rights Reserved.
Go to mobile version
Welcome Back!

Sign in to your account

Username or Email Address
Password

Lost your password?