By using this site, you agree to the Privacy Policy and Terms of Use.
Accept
SmartData CollectiveSmartData Collective
  • Analytics
    AnalyticsShow More
    football analytics
    The Role of Data Analytics in Football Performance
    9 Min Read
    data Analytics instagram stories
    Data Analytics Helps Marketers Make the Most of Instagram Stories
    15 Min Read
    analyst,women,looking,at,kpi,data,on,computer,screen
    What to Know Before Recruiting an Analyst to Handle Company Data
    6 Min Read
    AI analytics
    AI-Based Analytics Are Changing the Future of Credit Cards
    6 Min Read
    data overload showing data analytics
    How Does Next-Gen SIEM Prevent Data Overload For Security Analysts?
    8 Min Read
  • Big Data
  • BI
  • Exclusive
  • IT
  • Marketing
  • Software
Search
© 2008-23 SmartData Collective. All Rights Reserved.
Reading: Challenges of Working with Big Data: Beyond the 3Vs
Share
Notification Show More
Aa
SmartData CollectiveSmartData Collective
Aa
Search
  • About
  • Help
  • Privacy
Follow US
© 2008-23 SmartData Collective. All Rights Reserved.
SmartData Collective > Big Data > Data Mining > Challenges of Working with Big Data: Beyond the 3Vs
Data MiningData QualityData VisualizationData WarehousingSocial DataWorkforce Data

Challenges of Working with Big Data: Beyond the 3Vs

Venky Ganti
Last updated: 2015/07/16 at 10:50 PM
Venky Ganti
4 Min Read
SHARE

Among many challenges in working with big data, the 3V’s (Volume, Velocity, and Variety) have gotten a lot of attention. Googling yields many results worth reading. Almost all of these focus on technological challenges in managing and processing big data. In this post, I would like to highlight a different set issues that make working with big data challenging, even if the underlying infrastructure is admirably able to handle all three V’s.

Among many challenges in working with big data, the 3V’s (Volume, Velocity, and Variety) have gotten a lot of attention. Googling yields many results worth reading. Almost all of these focus on technological challenges in managing and processing big data. In this post, I would like to highlight a different set issues that make working with big data challenging, even if the underlying infrastructure is admirably able to handle all three V’s.

At Google, I had the opportunity to work within an amazing engineering team. I learnt various aspects of running services at scale as well as developing and launching compelling data products. I worked on the Dynamic Search Ads product which automates the AdWords campaign setup and optimization. Given an advertiser’s website, our goal was to mine relevant keywords, and for each keyword automatically create an advertisement (the ad text as well as the landing page). I worked with data from a variety of data sources, often for improving our product and sometimes for debugging issues.

We all know that Google organizes all of the information on the web and enables users to quickly find relevant information. But, how do many engineers feel about working with data at Google?

More Read

data mining

Data Mining Technology Helps Online Brands Optimize Their Branding

Data Visualization Boosts Business Scalability with Sales Mapping
Use this Strategic Approach to Maximize Your Data’s Value
Niche Data Tactics to Take Your Business to the Next Level
5 Best Practices for Extracting, Analyzing, and Visualizing Data

On the upside, they feel empowered in working with the rich data that Google collects from the huge amount of user activity on its property. Google’s data infrastructure ranks among the best out there. This is the place where many of the modern ideas of storing and processing “big data” originated. Combining these with a high calibre of engineers, a natural outcome is the creation of a massive number of information-rich derivative datasets.

On the down side, I think we could have been more effective and efficient with respect to finding and understanding data. Let me articulate some of the issues that contributed to these inefficiencies.

  • How do I find data that I can use for my current purpose? How do I understand the contents of a dataset after I find something?
  • Who do I ask for more information about the data? Has someone else used this data for a purpose similar to mine?
  • How do I debug unexpected data issues? Can upstream data changes explain such issues?
  • How do I set garbage collection policies for data I generate periodically?

In a couple of posts following this one, I will provide my experience around each of these questions, and how it impacted my efficiency besides raising the motivation bar for working with new data.

Venky Ganti July 16, 2015
Share This Article
Facebook Twitter Pinterest LinkedIn
Share

Follow us on Facebook

Latest News

Shutterstock Licensed Photo - 1051059293 | Rawpixel.com
QR Codes Leverage the Benefits of Big Data in Education
Big Data
football analytics
The Role of Data Analytics in Football Performance
Analytics Big Data Exclusive
smart home data
7 Mind-Blowing Ways Smart Homes Use Data to Save Your Money
Big Data
ai low code frameworks
AI Can Help Accelerate Development with Low-Code Frameworks
Artificial Intelligence

Stay Connected

1.2k Followers Like
33.7k Followers Follow
222 Followers Pin

You Might also Like

data mining
Data Mining

Data Mining Technology Helps Online Brands Optimize Their Branding

7 Min Read
data visualization for small business
Data Visualization

Data Visualization Boosts Business Scalability with Sales Mapping

7 Min Read
analyzing big data for its quality and value
Big Data

Use this Strategic Approach to Maximize Your Data’s Value

6 Min Read
niche data tactics for business success
Big Data

Niche Data Tactics to Take Your Business to the Next Level

6 Min Read

SmartData Collective is one of the largest & trusted community covering technical content about Big Data, BI, Cloud, Analytics, Artificial Intelligence, IoT & more.

AI chatbots
AI Chatbots Can Help Retailers Convert Live Broadcast Viewers into Sales!
Chatbots
AI and chatbots
Chatbots and SEO: How Can Chatbots Improve Your SEO Ranking?
Artificial Intelligence Chatbots Exclusive

Quick Link

  • About
  • Contact
  • Privacy
Follow US
© 2008-23 SmartData Collective. All Rights Reserved.
Go to mobile version
Welcome Back!

Sign in to your account

Lost your password?