We use cookies, including third-party cookies from Google to serve personalized ads through AdSense, to operate this site and understand how it is used. By continuing to browse, you accept this use. See our Privacy Policy and Terms of Use for details, including how to opt out of personalized advertising.
Accept
SmartData CollectiveSmartData Collective
  • Analytics
    AnalyticsShow More
    Flat editorial illustration: The article is about joining public web signals with internal business metrics so BI teams can inter
    Alternative Data in BI: Bringing Public Web Data Into Your Dashboards
    23 Min Read
    Flat editorial illustration: The article is about automating the collection, preparation, and delivery of marketing data across p
    From Data to Dashboards: 5 Marketing Reporting Automation Software Picks
    12 Min Read
    Flat editorial illustration: The article contrasts passive follower totals with a behavioral path from attention and trust to cli
    Creator Analytics Beyond Follower Count: Seven Metrics That Measure Audience Behavior
    11 Min Read
    Flat editorial illustration: The article connects AI and marketing analytics with managerial judgment grounded in consumer behavi
    How an Online MBA in Marketing Connects AI and Data Analytics
    11 Min Read
    What Kind of Problem-Solving Distinguishes Data Analysts From Software Engineers -- AI-generated illustration
    What Kind of Problem-Solving Distinguishes Data Analysts From Software Engineers
    7 Min Read
  • Big Data
  • BI
  • Exclusive
  • IT
  • Marketing
  • Software
Search
© 2008-25 SmartData Collective. All Rights Reserved.
Reading: Challenges of Working with Big Data: Beyond the 3Vs
Share
Notification
Font ResizerAa
SmartData CollectiveSmartData Collective
Font ResizerAa
Search
  • About
  • Help
  • Privacy
Follow US
© 2008-23 SmartData Collective. All Rights Reserved.
SmartData Collective > Big Data > Data Mining > Challenges of Working with Big Data: Beyond the 3Vs
Data MiningData QualityData VisualizationData WarehousingSocial DataWorkforce Data

Challenges of Working with Big Data: Beyond the 3Vs

Venky Ganti
Venky Ganti
4 Min Read
Challenges of Working with Big Data: Beyond the 3Vs
Illustration generated with FLUX.2 [klein 4B] via Cloudflare Workers AI.
SHARE

Among many challenges in working with big data, the 3V’s (Volume, Velocity, and Variety) have gotten a lot of attention. Googling yields many results worth reading. Almost all of these focus on technological challenges in managing and processing big data. In this post, I would like to highlight a different set issues that make working with big data challenging, even if the underlying infrastructure is admirably able to handle all three V’s.

At Google, I had the opportunity to work within an amazing engineering team. I learnt various aspects of running services at scale as well as developing and launching compelling data products. I worked on the Dynamic Search Ads product which automates the AdWords campaign setup and optimization. Given an advertiser’s website, our goal was to mine relevant keywords, and for each keyword automatically create an advertisement (the ad text as well as the landing page). I worked with data from a variety of data sources, often for improving our product and sometimes for debugging issues.

We all know that Google organizes all of the information on the web and enables users to quickly find relevant information. But, how do many engineers feel about working with data at Google?

On the upside, they feel empowered in working with the rich data that Google collects from the huge amount of user activity on its property. Google’s data infrastructure ranks among the best out there. This is the place where many of the modern ideas of storing and processing “big data” originated. Combining these with a high calibre of engineers, a natural outcome is the creation of a massive number of information-rich derivative datasets.

More Read

Dear Facebook, It’s Not You, It’s Us
Dear Facebook, It’s Not You, It’s Us
Avoiding Chaos, Losing Serendipity?
Does It Take a Scientist to Find Gold in Big Data?
How Hadoop Revolutionised IT
Comparing the Cost Continued…

On the down side, I think we could have been more effective and efficient with respect to finding and understanding data. Let me articulate some of the issues that contributed to these inefficiencies.

  • How do I find data that I can use for my current purpose? How do I understand the contents of a dataset after I find something?
  • Who do I ask for more information about the data? Has someone else used this data for a purpose similar to mine?
  • How do I debug unexpected data issues? Can upstream data changes explain such issues?
  • How do I set garbage collection policies for data I generate periodically?

In a couple of posts following this one, I will provide my experience around each of these questions, and how it impacted my efficiency besides raising the motivation bar for working with new data.

Share This Article
Facebook Pinterest LinkedIn
Share

Follow us on Facebook

Latest News

Flat editorial illustration: The article connects downstream report failures, data-quality gaps, and conflicting metric definitio
Signs Your Data Architecture Needs an Outside Review
Big Data Exclusive
Data-Driven Pultrusion: Best GFRP Rebar Machines & ROI
Data-Driven Pultrusion: Best GFRP Rebar Machines & ROI
Big Data Exclusive
Flat editorial illustration: The article contrasts tools for gathering fresh interviews with tools for analyzing existing qualita
10 AI Market Research Tools That Turn Qualitative Data into Insights
Artificial Intelligence Big Data Exclusive
Flat editorial illustration: The article compares proxy MCP servers by what an AI agent can do through them: manage proxy account
Proxy MCP Servers: 7 Options and What Each Lets an AI Agent Do
Artificial Intelligence Big Data Data Collection

Stay Connected

1.2KFollowersLike
33.7KFollowersFollow
222FollowersPin

You Might also Like

News from Washington: Teradata solutions for cloud computing and geospatial data
Data Warehousing

News from Washington: Teradata solutions for cloud computing and geospatial data

5 Min Read
The Enterprise Graph – From Connections To Customer Insights
AnalyticsBest PracticesBusiness IntelligenceBusiness RulesCRMData ManagementData MiningData QualityData VisualizationData WarehousingMarketingMarketing AutomationModelingPredictive Analytics

The Enterprise Graph – From Connections To Customer Insights

24 Min Read
Google's Year in Local Search
Data Visualization

Google’s Year in Local Search

3 Min Read
What the "Small Data" Revolution Means for Marketers
Big DataBusiness IntelligenceCRMData MiningExclusiveMarketing

What the “Small Data” Revolution Means for Marketers

6 Min Read

SmartData Collective is one of the largest & trusted community covering technical content about Big Data, BI, Cloud, Analytics, Artificial Intelligence, IoT & more.

Artificial Intelligence for eCommerce: A Closer Look
Artificial Intelligence for eCommerce: A Closer Look
Artificial Intelligence
The Art of Conversation: Enhancing Chatbots with Advanced AI Prompts
The Art of Conversation: Enhancing Chatbots with Advanced AI Prompts
Chatbots

Quick Link

  • About
  • Contact
  • Privacy
Follow US
© 2008-26 SmartData Collective. All Rights Reserved.
Welcome Back!

Sign in to your account

Username or Email Address
Password

Lost your password?