By Michael Lieberman
Many recent internet posts (e.g. Twitter, LinkedIn) have advanced the claim that “Big Data will replace traditional survey research.” The authors advocate extensive use of available data, data processing platforms, or do-it-yourself tools such as Survey Monkey to allow corporations to conduct their own research. This idea is quite misguided. Online panels did not replace traditional phone data collection, or common market research methods like focus groups, laddering interviews, mall intercepts or taste kitchens. Just try to test a new bagel on the internet, for example. We suggest that Big Data will enhance the reach of the survey research industry, not replace it.
What is Big Data?
Surprisingly this question is often asked in professional circles. While Big Data is the subject of much speculation, we have often found that research suppliers, corporate researchers, and C-suite marketing executives do not fully understand Big Data. The fuzzy thinking goes something like this – It is something big; it has to do with data. The details are open to interpretation. In order to look more deeply into this question, consider the following. When one performs a search on Google, makes a comment on Facebook, or tweets, he creates data, but this is only part of the data creation explosion. A purchase at Walmart.com or signs-up for Netflix also creates data. An email to the SanDiego.org, a booking on Expedia, taking out a mortgage, filing income taxes, or making an insurance claim creates data.
Big data falls into two distinct categories:
Data Used for Predictive Analytics
In this category corporations collect and warehouse the data for analytic purposes. A mortgage company stores a list of good loan candidates. An insurance company lists possible fraud sources. Amazon predicts new customer product interests based on past behavior. As Eric Siegel of Predictive Analytics World states, Predictive Analytics holds “the power to predict who will click, buy, lie, or die.”
Social Network Data
The social networks Facebook, Twitter, LinkedIn, and Flickr create unstructured data. The flow of comments and information between participants is free. The tracking of such information and extracting meaning from it has spawned control rooms in companies like Nike and Starbucks to monitor, or if possible, control the brand conversations. New fields such as Social Media Network Analysis and Sentiment Analysis (natural language processing) are springing up. Large advertising firms are using software for Social Media Monitoring to tame the unstructured data universe. Web analytics firms are popping up everywhere.
Big Data and Marketing Research
Over our more than 25 years in the marketing research industry, we have come to understand that more and more marketing research projects focus on a specific marketing problem, for example:
- How to segment shoppers at Target
- How to construct an optimal product using conjoint analysis
- How much should a company charge for a product before sales begin to drop sharply; which message most resonates with voters or sub-segments of voters
One important reason that Big Data will not replace marketing research is its lack of specificity when addressing these narrow project goals. While one can get a general sense of what is being said about a brand using Sentiment Analysis on tweets or Facebook posts, or by extracting data from an immense customer database (and produce an impressive report using data mining techniques), a focused marketing goal cannot be clearly addressed using only such “buckshot” data harvesting methods. The more focused marketing questions that can only be addressed through focused methods like surveys. This sort of research is not going to disappear any time soon, and neither will the marketing research industry.
Despite its limitations, we suggest that the two aspects of Big Data can enhance marketing research. Predictive analytics, particularly in those companies that hold large customer information warehouses, can append information to survey data to add segmentation value. In addition, survey researchers can use data mining from such warehouses to create custom reports or segmentations. Conventional data mining tools, such as Regression Analysis, CHAID or CART Trees, or Neural Networks, are employed by both Predictive Analytics and marketing research. Within the social networking space, Sentiment Analysis, Text Analytics, and Social Network Visualization are valuable tools to add texture to a full-service marketing research report. In some ways we see this as a mixture of the qualitative/quantitative relationship found in marketing research. In many ways this is already being accomplished with online chats performed after a quantitative survey. Tweets, Flickr groups, and Facebook can add valuable insights to a well constructed branding project—but cannot fully replace it.
In response to the blogosphere premise “that Big Data can replace survey research” our retort is that Big Data along with analytic techniques augments marketing research. Together, “Big Data” and marketing research represent a formidable toolkit in the hands of those who know how and when to exploit both to their full potential.