By using this site, you agree to the Privacy Policy and Terms of Use.
Accept
SmartData CollectiveSmartData Collective
  • Analytics
    AnalyticsShow More
    analyst,women,looking,at,kpi,data,on,computer,screen
    What to Know Before Recruiting an Analyst to Handle Company Data
    6 Min Read
    AI analytics
    AI-Based Analytics Are Changing the Future of Credit Cards
    6 Min Read
    data overload showing data analytics
    How Does Next-Gen SIEM Prevent Data Overload For Security Analysts?
    8 Min Read
    hire a marketing agency with a background in data analytics
    5 Reasons to Hire a Marketing Agency that Knows Data Analytics
    7 Min Read
    predictive analytics for amazon pricing
    Using Predictive Analytics to Get the Best Deals on Amazon
    8 Min Read
  • Big Data
  • BI
  • Exclusive
  • IT
  • Marketing
  • Software
Search
© 2008-23 SmartData Collective. All Rights Reserved.
Reading: Why Variety Is the Unsolved Problem in Big Data
Share
Notification Show More
Aa
SmartData CollectiveSmartData Collective
Aa
Search
  • About
  • Help
  • Privacy
Follow US
© 2008-23 SmartData Collective. All Rights Reserved.
SmartData Collective > Big Data > Data Quality > Why Variety Is the Unsolved Problem in Big Data
Big DataData QualityIT

Why Variety Is the Unsolved Problem in Big Data

MaheshKumar1
Last updated: 2013/10/30 at 8:00 AM
MaheshKumar1
6 Min Read
Image
SHARE

ImageThe term “big data” is thrown around rather loosely today. To apply more structure, Gartner classifies big data projects by the “3 V’s” – volume, velocity, and variety in its IT glossary:

ImageThe term “big data” is thrown around rather loosely today. To apply more structure, Gartner classifies big data projects by the “3 V’s” – volume, velocity, and variety in its IT glossary:

“Big data is high-volume, high-velocity and high-variety information assets that demand cost-effective, innovative forms of information processing for enhanced insight and decision making.” 

Technology advances have helped us enormously in dealing with the first two attributes – volume and velocity. Advances in storage technologies have brought down costs of storing all of that data, and technologies like Apache™ Hadoop® help companies assemble the processing power by distributing computing across inexpensive, redundant components.  

More Read

data breaches

How Hospital Security Breaches Devastate Local Communities

What to Know Before Recruiting an Analyst to Handle Company Data
Tackling Bias in AI Translation: A Data Perspective
Data Ethics: Safeguarding Privacy and Ensuring Responsible Data Practices
8 Crucial Tips to Help SMEs Guard Against Data Breaches

But the issue of data variety remains much more difficult to solve programmatically. Instead, we call on experts in big data applications in specific domains. As a result, many big data initiatives remain constrained by the skills of the people available to work on them. And this challenge is keeping the industry from realizing the full potential of big data in diverse fields.

The symptom of the problem: Services spending

If you look at recent history, most technology innovations follow a pattern. As technologies evolve, eventually the differentiation – and money – flows to the software. Marc Andreessen famously outlined this pattern with his “Software is Eating the World” manifesto in the Wall Street Journal in 2001.

Now look at big data spending today – according to recent numbers from Gartner, spending on services outweighs spending on software by a ratio of nine to one*. Even if you account for the fact that much of the software is open source, that’s still a lot of spending on services. In fact, Gartner projects that services spending will reach more than $40 billion by 2016. 

Services spending is a symptomatic of a larger problem that cannot easily be solved with software. If it was easily solvable, someone would have figured it out, given the amount of spending going into services today. I think that the problem lies in data variety – the sheer complexity of the multitude of data sources, good and bad data mixed together, multiple formats, multiple units and the list goes on. As a result of this unsolved problem, we’re grooming a large field of specialists with proficiency in specific domains, such as marketing data, social media data, telco data, etc. And we’re paying those people well, because their skills are both valuable and relatively scarce. 

Drilling down into the data variety problem

When META Group (now Gartner) analyst Doug Laney first wrote about the big data definition in 2001, he discussed the ‘variety’ part of the big data challenge as referring to data formats, structures and semantics. 

More than a decade later, the online world is a much larger, more interconnected and complex place. The sheer variety of available data for analysis has grown exponentially since that definition in 2001. To paraphrase Hamlet, “There are more data types in cyberspace than are dreamt of in your definitions.” And with the coming Internet of Things, the variety of data will continue to grow as the devices collecting and sending data proliferate.

Data variety and context

When it comes to data variety, a large part of the challenge lies in putting the data into the right context. Nothing exists in isolation in today’s networked world as most of the big data available for analysis is linked to outside entities and organizations. Making sense of the context takes time and human understanding and that slows everything down.

Today, it falls to people to address the larger problem of variety by making sense of and adding context to the diverse data types and sources (hence the large services spending cited above).  These people need both domain expertise, to understand the context of the data, and big data skills, to understand how to use the data.

Until we come up with a scalable and viable way to address the “high-variety” part of the big data challenge, we’ll continue to rely on people and services. This will keep the cost of big data initiatives high and limit their applications in new environments, where the potential for new insights may be high, but the budget simply doesn’t exist to apply big data disciplines.

*Gartner, “Big Data Drives Rapid Changes in Infrastructure and $232 Billion in IT Spending Through 2016,” October 2012

image: variety/shutterstock

MaheshKumar1 October 30, 2013
Share This Article
Facebook Twitter Pinterest LinkedIn
Share

Follow us on Facebook

Latest News

data breaches
How Hospital Security Breaches Devastate Local Communities
Policy and Governance
analyst,women,looking,at,kpi,data,on,computer,screen
What to Know Before Recruiting an Analyst to Handle Company Data
Analytics
data perspective
Tackling Bias in AI Translation: A Data Perspective
Big Data
Data Ethics: Safeguarding Privacy and Ensuring Responsible Data Practices
Data Ethics: Safeguarding Privacy and Ensuring Responsible Data Practices
Best Practices Big Data Data Collection Data Management Privacy

Stay Connected

1.2k Followers Like
33.7k Followers Follow
222 Followers Pin

You Might also Like

data breaches
Policy and Governance

How Hospital Security Breaches Devastate Local Communities

7 Min Read
analyst,women,looking,at,kpi,data,on,computer,screen
Analytics

What to Know Before Recruiting an Analyst to Handle Company Data

6 Min Read
data perspective
Big Data

Tackling Bias in AI Translation: A Data Perspective

9 Min Read
Data Ethics: Safeguarding Privacy and Ensuring Responsible Data Practices
Best PracticesBig DataData CollectionData ManagementPrivacy

Data Ethics: Safeguarding Privacy and Ensuring Responsible Data Practices

7 Min Read

SmartData Collective is one of the largest & trusted community covering technical content about Big Data, BI, Cloud, Analytics, Artificial Intelligence, IoT & more.

AI chatbots
AI Chatbots Can Help Retailers Convert Live Broadcast Viewers into Sales!
Chatbots
giveaway chatbots
How To Get An Award Winning Giveaway Bot
Big Data Chatbots Exclusive

Quick Link

  • About
  • Contact
  • Privacy
Follow US
© 2008-23 SmartData Collective. All Rights Reserved.
Go to mobile version
Welcome Back!

Sign in to your account

Lost your password?