Douglas Merrill, former CIO/VP of Engineering at Google, writes in Forbes about using the R language for data analysis:
Most folks with math-oriented graduate degrees will have written something in R, a non-commercial option for your big data analysis. So, great graduates from great graduate schools know great tools.
His post is titled ‘R Is Not Enough For “Big Data”‘, and you might be surprised to learn that I agree that title, although for a different reason. Douglas’s point — and it’s a valid one — is that simply pumping data through any software tool, without an understanding of the problem you’re trying to solve and how statistical models apply to it, can lead to getting the wrong answers to the wrong questions:
If you ask the wrong question, you will be able to find statistics that give answers that are simply wrong (or, at best, misleading).
On net, having a degree in math, economics, AI, etc., isn’t enough. Tool expertise isn’t enough. You need experience in solving real world problems, because there are a lot of importat limitations to the statistics that you learned in school. Big data isn’t about bits, it’s about talent.
This is a great illustration of why the data science process is a valuable one for extracting information from Big Data, because it combines tool expertise with statistical expertise and the domain expertise required to understand the problem and the data applicable to it. He’s right that you need data science talent and software to solve problems with Big Data … and having software like R that supports the exploratory nature of the data science process is also critical.
But I also agree with the title for a different, technical reason: the R software is just one piece of software ecosystem — an analytics stack, if you will — of tools used to analyze Big Data. For one thing R isn’t a data store in its own right: you also need a data layer where R can access structured and unstructured data for analysis. (For example, see how you can use R to extract data from Hadoop in the slides from today’s webinar by Antonio Piccolboni.) At the analytics layer, you need statistical algorithms that work with Big Data, like those in Revolution R Enterprise. And at the presentation layer, you need the ability to embed the results of the analysis in reports, BI tools, or data apps.
So yes, Douglas is right: you need more than just R for Big Data. You also need a data layer, an analytics layer, and a presentation layer (all of which supports Big Data) … and you need Data Science skills to make sure you’re asking the right questions and getting appropriate answers.
Forbes: R Is Not Enough For “Big Data”
