Graphing real-time foreclosure data: Data Mashups in R

June 23, 2009
60 Views

The latest in the O’Reilly “Short Cuts” series, and the first devoted to R, is Data Mashups in R. Written by Jeremy Leipzig and Ziao-Yi Lu, this 30-page article is an excellent and very practical example of integrating messy data from varied sources, using R or REvolution R.

It’s not designed as a manual-style introduction to R. But while working through the fully-detailed example it presents, even programmers unfamiliar to R will get a good sense of the practical capabilities of R when working with real-life data sources. “Learning by doing” is a great way to bootstrap your knowledge of any language, and a concrete, practical example like this really helps with that process.

The example used is a particularly timely one: how to automate the process of downloading foreclosure data from a public website, and presenting it in graphical form, like this:

Philly-foreclosures

In practical terms, creating this graphic necessitates the integration of data from various sources:
  • Downloading an HTML file of foreclosed addresses from a public web-site (with download.file) and extracting addresses in messy formats (with grep);
  • Downloading geolocation data from a Yahoo web-service and parsing the

The latest in the O’Reilly “Short Cuts” series, and the first devoted to R, is Data Mashups in R. Written by Jeremy Leipzig and Ziao-Yi Lu, this 30-page article is an excellent and very practical example of integrating messy data from varied sources, using R or REvolution R.

It’s not designed as a manual-style introduction to R. But while working through the fully-detailed example it presents, even programmers unfamiliar to R will get a good sense of the practical capabilities of R when working with real-life data sources. “Learning by doing” is a great way to bootstrap your knowledge of any language, and a concrete, practical example like this really helps with that process.

The example used is a particularly timely one: how to automate the process of downloading foreclosure data from a public website, and presenting it in graphical form, like this:

Philly-foreclosures

In practical terms, creating this graphic necessitates the integration of data from various sources:
  • Downloading an HTML file of foreclosed addresses from a public web-site (with download.file) and extracting addresses in messy formats (with grep);
  • Downloading geolocation data from a Yahoo web-service and parsing the XML result (with xmlTreeParse);
  • Downloading an ESRI shape file of Philadelphia and its census tracts, and plotting a map (with the maptools package);
  • Matching the individual addresses to census tracts and counting the number of foreclosures in each (with plotPolys).
One of the great things about the article is that it’s not presented as a theoretical exercise where everything just works the first time. It takes pains to cover not just how to accomplish the task itself, but also how to deal with problems you’re likely to encounter along the way (especially if you adapt the methods to your own needs). Practical advice is given on how to deal with connection problems caused by firewalls and proxies, how to deal gracefully with the failure of servers to respond (using tryCatch), and how to detect error messages from the Yahoo web service when it fails to recognize addresses. The authors also let you know where they found the information of how to deal with such problems, so it also serves as a guide for finding help resources in R.

My only criticisms are minor ones. The article leads the user through downloading Census data, matching it to the tracts, and performing some simply summary statistics and exploratory graphics, but stops short of any actual statistical analysis. And cutting-and-pasting R code from a PDF document is a pain: there’s no associated script file (as far as I know).

But for experienced R users looking for tips on integrating messy data sources from the Web, or for programmers new to R looking for a practical example to work through as an introduction to R, Data Mashups in R is well worth $4.99 to download.

O’Reilly Short Cuts: Data Mashups in R

Link to original post

You may be interested

Big Data Revolution in Agriculture Industry: Opportunities and Challenges
Analytics
25 views
Analytics
25 views

Big Data Revolution in Agriculture Industry: Opportunities and Challenges

Kayla Matthews - July 24, 2017

Big data is all about efficiency. There are many types of data available, and many ways to use that information.…

How SAP Hana is Driving Big Data Startups
Big Data
298 shares3,195 views
Big Data
298 shares3,195 views

How SAP Hana is Driving Big Data Startups

Ryan Kh - July 20, 2017

The first version of SAP Hana was released in 2010, before Hadoop and other big data extraction tools were introduced.…

Data Erasing Software vs Physical Destruction: Sustainable Way of Data Deletion
Data Management
154 views
Data Management
154 views

Data Erasing Software vs Physical Destruction: Sustainable Way of Data Deletion

Manish Bhickta - July 20, 2017

Physical Data destruction techniques are efficient enough to destroy data, but they can never be considered eco-friendly. On the other…