I was recently invited to give keynote addresses at Australia’s AMSRS conference and the USA’s CRC conference about Big Data. It’s a topic that has gotten a massive amount of attention over the last few years particularly within the marketing research community.
Big data is commonly associated with “The Three Vs,” specifically volume, velocity, and variety. Although big data is often considered to be the domain of data scientists, these three characteristics are near and dear to the heart of market researchers.
For instance, within a research panel company such as Peanut Labs, the volume of data generated is simply massive. For every person who joins the panel, all of their demographic data is recorded in order to determine which surveys are most relevant to them. And when they complete surveys, information about the day and time are retained in order to keep track of which panelists are engaged or not engaged in the process. Across millions of panelists, this amounts to hundreds of millions of data points that must be analyzed using specialized statistical software such as SAS or SQL. It’s certainly more than you’d like to work with in an excel worksheet!
The velocity of the data generated is also incredible. Every time a new person joins the panel, a record of their many demographic details is saved. Every time a panelist begins a survey, completes a survey, or gives up part way through a survey, another record is entered into the panel database. Across millions of panelists, that leads to multiple records being added to many different databases, from incentive databases to data quality databases to survey invitation databases, every single second.
And finally, consider variety. The databases used for research panel incorporate such a wide variety of data. Dates, times, birthdays, geography, usernames, survey lengths, incentives values, and so much more. When it comes to the variety of data, we’ve got that covered too.
Once you add an in depth knowledge of statistics, database coding, and domain knowledge, market researchers have a pretty good handle of big data. Perhaps it’s time to realize that Data Scientists are simple Market Researchers by another name.
