Cookies help us display personalized product recommendations and ensure you have great shopping experience.

By using this site, you agree to the Privacy Policy and Terms of Use.
Accept
SmartData CollectiveSmartData Collective
  • Analytics
    AnalyticsShow More
    data analytics and truck accident claims
    How Data Analytics Reduces Truck Accidents and Speeds Up Claims
    7 Min Read
    predictive analytics for interior designers
    Interior Designers Boost Profits with Predictive Analytics
    8 Min Read
    image fx (67)
    Improving LinkedIn Ad Strategies with Data Analytics
    9 Min Read
    big data and remote work
    Data Helps Speech-Language Pathologists Deliver Better Results
    6 Min Read
    data driven insights
    How Data-Driven Insights Are Addressing Gaps in Patient Communication and Equity
    8 Min Read
  • Big Data
  • BI
  • Exclusive
  • IT
  • Marketing
  • Software
Search
© 2008-25 SmartData Collective. All Rights Reserved.
Reading: Challenges of Working with Big Data: Beyond the 3Vs
Share
Notification
Font ResizerAa
SmartData CollectiveSmartData Collective
Font ResizerAa
Search
  • About
  • Help
  • Privacy
Follow US
© 2008-23 SmartData Collective. All Rights Reserved.
SmartData Collective > Big Data > Data Mining > Challenges of Working with Big Data: Beyond the 3Vs
Data MiningData QualityData VisualizationData WarehousingSocial DataWorkforce Data

Challenges of Working with Big Data: Beyond the 3Vs

Venky Ganti
Venky Ganti
4 Min Read
SHARE

Among many challenges in working with big data, the 3V’s (Volume, Velocity, and Variety) have gotten a lot of attention. Googling yields many results worth reading. Almost all of these focus on technological challenges in managing and processing big data. In this post, I would like to highlight a different set issues that make working with big data challenging, even if the underlying infrastructure is admirably able to handle all three V’s.

Among many challenges in working with big data, the 3V’s (Volume, Velocity, and Variety) have gotten a lot of attention. Googling yields many results worth reading. Almost all of these focus on technological challenges in managing and processing big data. In this post, I would like to highlight a different set issues that make working with big data challenging, even if the underlying infrastructure is admirably able to handle all three V’s.

At Google, I had the opportunity to work within an amazing engineering team. I learnt various aspects of running services at scale as well as developing and launching compelling data products. I worked on the Dynamic Search Ads product which automates the AdWords campaign setup and optimization. Given an advertiser’s website, our goal was to mine relevant keywords, and for each keyword automatically create an advertisement (the ad text as well as the landing page). I worked with data from a variety of data sources, often for improving our product and sometimes for debugging issues.

We all know that Google organizes all of the information on the web and enables users to quickly find relevant information. But, how do many engineers feel about working with data at Google?

More Read

Can you learn data mining in undergraduate or graduate school?
A Big Data Cheat Sheet: What Executives Want to Know
Do You Have Any Rights in the Age of Big Data Analytics?
How the Healthcare Industry Can Save Big with Big Data
Top 20 Data Science Skills

On the upside, they feel empowered in working with the rich data that Google collects from the huge amount of user activity on its property. Google’s data infrastructure ranks among the best out there. This is the place where many of the modern ideas of storing and processing “big data” originated. Combining these with a high calibre of engineers, a natural outcome is the creation of a massive number of information-rich derivative datasets.

On the down side, I think we could have been more effective and efficient with respect to finding and understanding data. Let me articulate some of the issues that contributed to these inefficiencies.

  • How do I find data that I can use for my current purpose? How do I understand the contents of a dataset after I find something?
  • Who do I ask for more information about the data? Has someone else used this data for a purpose similar to mine?
  • How do I debug unexpected data issues? Can upstream data changes explain such issues?
  • How do I set garbage collection policies for data I generate periodically?

In a couple of posts following this one, I will provide my experience around each of these questions, and how it impacted my efficiency besides raising the motivation bar for working with new data.

Share This Article
Facebook Pinterest LinkedIn
Share

Follow us on Facebook

Latest News

data analytics and truck accident claims
How Data Analytics Reduces Truck Accidents and Speeds Up Claims
Analytics Big Data Exclusive
predictive analytics for interior designers
Interior Designers Boost Profits with Predictive Analytics
Analytics Exclusive Predictive Analytics
big data and cybercrime
Stopping Lateral Movement in a Data-Heavy, Edge-First World
Big Data Exclusive
AI and data mining
What the Rise of AI Web Scrapers Means for Data Teams
Artificial Intelligence Big Data Exclusive

Stay Connected

1.2kFollowersLike
33.7kFollowersFollow
222FollowersPin

You Might also Like

Find yourself a safer place to swim or fish in the Bay Area

4 Min Read

Where the Newburyport City Council is Online

5 Min Read

Google: Find Similar Images

3 Min Read

Social Searching Everything, For Everyone: Product Review

4 Min Read

SmartData Collective is one of the largest & trusted community covering technical content about Big Data, BI, Cloud, Analytics, Artificial Intelligence, IoT & more.

ai in ecommerce
Artificial Intelligence for eCommerce: A Closer Look
Artificial Intelligence
data-driven web design
5 Great Tips for Using Data Analytics for Website UX
Big Data

Quick Link

  • About
  • Contact
  • Privacy
Follow US
© 2008-25 SmartData Collective. All Rights Reserved.
Go to mobile version
Welcome Back!

Sign in to your account

Username or Email Address
Password

Lost your password?