Cookies help us display personalized product recommendations and ensure you have great shopping experience.

By using this site, you agree to the Privacy Policy and Terms of Use.
Accept
SmartData CollectiveSmartData Collective
  • Analytics
    AnalyticsShow More
    sales and data analytics
    How Data Analytics Improves Lead Management and Sales Results
    9 Min Read
    data analytics and truck accident claims
    How Data Analytics Reduces Truck Accidents and Speeds Up Claims
    7 Min Read
    predictive analytics for interior designers
    Interior Designers Boost Profits with Predictive Analytics
    8 Min Read
    image fx (67)
    Improving LinkedIn Ad Strategies with Data Analytics
    9 Min Read
    big data and remote work
    Data Helps Speech-Language Pathologists Deliver Better Results
    6 Min Read
  • Big Data
  • BI
  • Exclusive
  • IT
  • Marketing
  • Software
Search
© 2008-25 SmartData Collective. All Rights Reserved.
Reading: ggplot2 for Big Data
Share
Notification
Font ResizerAa
SmartData CollectiveSmartData Collective
Font ResizerAa
Search
  • About
  • Help
  • Privacy
Follow US
© 2008-23 SmartData Collective. All Rights Reserved.
SmartData Collective > Big Data > Data Visualization > ggplot2 for Big Data
Data VisualizationR Programming Language

ggplot2 for Big Data

DavidMSmith
DavidMSmith
4 Min Read
SHARE

(Hadley Wickham, author of ggplot2 and several other R packages, guest blogs today about forthcoming big-data improvements to his R graphics package — ed.) 

(Hadley Wickham, author of ggplot2 and several other R packages, guest blogs today about forthcoming big-data improvements to his R graphics package — ed.) 

Hi! I’m Hadley Wickham and I’m guest posting on the Revolutions blog to give you a taste of some of the visualisation work that my research team and I worked on this summer. This work has been generously funded by Revolution Analytics and while, as you’ll see, it works particularly well with RevoScaleR, it’s also contributing to changes that will help all ggplot2 users.

More Read

Wordtree for Visual Text Exploration
Data Driven: 5 Ways Automakers Use Big Data to Improve Their Products
Answering the real questions in data analysis
The IM Strategy Stack
What is Big Data?

This summer three undergrads, James Rigby, Jonathan Stewart, Hyun Bin Kang and one grad student, Ben White, have been working on the answer to an important question: how can we make a scatterplot work when you have hundreds of millions of points? Scatterplots are one of the most important tools of exploratory data analysis, but they start to break down even with relatively small datasets because of overplotting: you can’t tell how many points are plotted at each location. They also get slower and slower the more points you try to draw.

The answer to both these problems is relatively simple: instead of plotting the raw data, plot densities, or distributions. These can generated simply (by binning and counting the number of points in each bin), or with more sophistication (by smoothing the bin counts to get a kernel density estimate). RevoScaleR makes this process incredibly fast: you can bin tens of millions observations in a few seconds on commodity hardware, and a kernel density estimate only takes a fraction more to compute.

Once you have the density, what can you do with it? The following plots show two of the ideas that we came up with. The examples show the diamonds data set from ggplot2, but the beauty of these techniques is that they’ll work regardless of how much data you have – the extra complexity is taken care of in density computation stage.

A scatterplot of depth and table coloured by the z dimension is uninformative because of the extreme amount of overplotting:

Continuous-scatter

One way to make it better is bin and smooth in 3d and then plot the conditional distribution of z at multiple values of depth and table.

Continuous-density
The shape shows the distribution of z, and the colour shows the total density at that location — higher values mean there are more data points in that location. This plot reveals much more than the previous one: most of the data points are concentrated near depth 56 and table 60 where the distribution of z is skewed towards smaller values. As depth increases, the average value of z also seems to increase.

A scatterplot of carat vs. price coloured by the colour of the diamond:

Discrete-scatter
There is some hint that J colours are relatively cheaper (for a given size they have lower prices) but it’s hard to see anything else because of the overplotting. Binning the data and displaying the distribution of colour in each bin makes the important patterns much easier to see.

Discrete-density
Colours D, E and F are more expensive side, and H, I and particularly J are cheaper. Only bins containing more than 100 points are included to avoid drawing the eye to regions with little data.

I’m currently with working another student, Yue Hu, to turn our research into a robust R package.

Share This Article
Facebook Pinterest LinkedIn
Share

Follow us on Facebook

Latest News

sales and data analytics
How Data Analytics Improves Lead Management and Sales Results
Analytics Big Data Exclusive
ai in marketing
How AI and Smart Platforms Improve Email Marketing
Artificial Intelligence Exclusive Marketing
AI Document Verification for Legal Firms: Importance & Top Tools
AI Document Verification for Legal Firms: Importance & Top Tools
Artificial Intelligence Exclusive
AI supply chain
AI Tools Are Strengthening Global Supply Chains
Artificial Intelligence Exclusive

Stay Connected

1.2kFollowersLike
33.7kFollowersFollow
222FollowersPin

You Might also Like

10 Don’ts for Delivering Mobile Business Intelligence

14 Min Read

Independence Day Infographics Explode onto the Data Scene

3 Min Read

Salesforce Chatter: Facebook for the enterprise?

2 Min Read

Quick Strata update

2 Min Read

SmartData Collective is one of the largest & trusted community covering technical content about Big Data, BI, Cloud, Analytics, Artificial Intelligence, IoT & more.

AI chatbots
AI Chatbots Can Help Retailers Convert Live Broadcast Viewers into Sales!
Chatbots
ai in ecommerce
Artificial Intelligence for eCommerce: A Closer Look
Artificial Intelligence

Quick Link

  • About
  • Contact
  • Privacy
Follow US
© 2008-25 SmartData Collective. All Rights Reserved.
Go to mobile version
Welcome Back!

Sign in to your account

Username or Email Address
Password

Lost your password?