PureSpectrum - Schedule A Demo
Our new GreenBook Directory site is live!
COVID-19 guidance, tips, analysis - access full coverage here

Why Survey?

Survey research has been declared dead before, yet it is still with us. But has its death been exaggerated once more? Are traditional survey data utilized to their fullest? Are the New Data in reality an opportunity for survey research?

survey puzzle

Editor’s Note: It’s easy to assume that with all of the new tools and data sources coming into play in the insights space that the stalwart survey is no longer relevant. I am not in that camp, and judging by the success of Survey Monkey, Google Consumer Surveys, and myriad other variations on survey models in the marketplace many others see the value of this tool. That said, I am skeptical that the survey will remain the primary tool used in the research process; we are witnessing a transition period now where new approaches are augmenting survey data and I fully expect at the end of the transition for the roles to be reversed and surveys will be used to augment and fill in the data gaps when other data sources are primary.

Today’s post by first time GreenBook Blog author Kevin Gray is a wonderful reminder of how useful the survey is and will remain to be in the foreseeable future. It’s an honest appraisal of the pros and cons of the technique and a great primer on how to think about the role of surveys in the new insights era we are in. It’s worth noting that Kevin is a 25 year veteran of the space, a well respected researcher, and a voice of reason on many discussion threads in the various MR-focused LinkedIn groups. He’s someone I respect immensely and I am thrilled to share this great first post with you; it moves the dialogue forward for us all.

Here is Kevin’s post in defense of the humble survey. Enjoy!

By Kevin Gray

Imagine a world in which every consumer’s shopping and purchase behavior, Social Media and other public online activity are monitored in real time, and that there is an ever-expanding data file for each and every consumer available to marketers. 

At some point in the not-too-distant future few technological hurdles may remain in the way of this marketing utopia.  There are of course privacy concerns that might prevent this (arguably dystopian) world from coming into existence.  But, for the sake of discussion, let’s assume that world is here.   Would you, as a marketer, still need to survey consumers? 

Yes!  One example would be when a product or service is not yet on the market.  For instance, you may have a product prototype, or just an idea about a product, and you would like detailed consumer reactions about it.  Concept screening is another example.   The requisite data are simply not “out there” waiting to be collected. 

Back to the Present…

In today’s real world vast amounts of data from sources other than consumer surveys are already available to marketers.  Let’s think about customer records for a moment.  Customer records are “hard” data on individual customers.  They may include detailed transaction and demographic information.  These data are core to certain specialized analytics, for example market basket analysis and recommender systems.  Many kinds of customer records have been mined for years –  this is not a new development and predates Customer Relationship Management.

A limitation of analytics which only make use of customer records is that intangible but important variables such as brand awareness, image and attitudinal data, are absent.  To a degree, these can be inferred from past behavior and demographic data that may be attached to each customer record.  Social Media, call center records and other external data sometimes can also be leveraged.  However, specific information crucial for brand building and other essentials of marketing for individual consumers is mostly lacking.  Consumer surveys can be used to plug these gaps. 

“But Surveys Are Flawed!”

Consumer surveys are not without their critics, as any marketing researcher knows.  One concern is that consumers often can’t answer questions important to marketers with great precision.  It’s difficult to remember exactly how many times we bought Brand X in the past three months.  We also may not really know why we bought Brand X.  Honest answers to psychographic questions can be challenging and socially-desirable responses can contaminate the data.  Respondents, understandably, may also be reluctant to reveal information they regard as personal, such as income.

These are long-standing issues in survey research.  They can be seriously exacerbated by substandard research design, amateurish questionnaires and poor fieldwork execution.  A flawed questionnaire administered to the wrong consumers, combined with a low response rate is a marketing research nightmare. 

Why do surveys go wrong?  There can be many reasons.  The respondent group we interview may be too narrowly-defined because of mistaken presumptions about consumers of the category, for instance.  Questions may include legacy items no longer meaningful or items made up on the fly that are ambiguous or even nonsensical to many consumers.  In the case of multi-country studies, psychographic items developed with U.S. consumers in mind, for instance, may be irrelevant to consumers in other cultures. 

Survey Data Analytics

That all said, there is a legal adage stating that “hard cases make bad law” which may bear on this discussion.  Irretrievably shoddy surveys are not the rule, fortunately.  Most survey research is valuable to the end users and does fill gaps left by other data sources.  Let’s elaborate a bit.  Marketers need to drill down and look at the interrelationships among many variables.  These analyses are especially useful when they are able to tie together specific kinds of data for individual consumers.  Here are a few examples of how marketers can use consumer survey data (and also move beyond standard Field & PowerPoint studies).

·         Post hoc segmentation is widely-used to identify consumer groups for targeting purposes.  With recent advances in statistics and machine learning, we are now less bound to the customary “Cluster & Pray” approach to segmentation, in which cluster analysis is applied to attitudinal data and candidate clusters cross tabulated against demographics and key consumption behaviors…in the hope they will connect meaningfully.

·         Generating a massive number of crosstabs, besides being costly and inefficient, risks finding sexy results that are really flukes.  Post hoc segmentation is an alternative.  It can also serve as an exploratory analysis tool to develop knowledge and insights about the various ways different kinds of consumers behave.  Used this way, segmentation can be thought of as a kind of multivariate crosstab. 

·         In addition to post hoc segmentation, multivariate analysis can be used to profile pre-defined consumer groups, such as heavy, medium and light category users.  A form of data mining, this latter approach to is sometimes called a priori segmentation.  There are now many methods at our disposal for these sorts of analyses.

·         Key driver analysis comes into play when the objective is to identify important associations between predictor variables, e.g., product attribute or satisfaction ratings, and one or more target variables, such as purchase intention for a brand or overall satisfaction with customer service.  It’s typically done with all predictors – independent variables – considered simultaneously, and when performed competently is superior to simple correlation analysis or multiple cross tabs.

·         Perceptual mapping gives us rich insights into how consumers see the market.  Consumer segments can also be “mapped.”  Mapping comes in many flavors.  Some methods, like Correspondence Analysis, are particularly useful for revealing brand image profiles and reducing brand size effect, whereby the big brands dominate the map.

·         New Product Development (NPD) was briefly referred at the outset.  Concept screening, product testing, simulated test marketing and conjoint studies are methods commonly employed in NPD.  All require consumer surveys.

It’s not Either/Or (with apologies to Søren Kierkegaard)

Luckily, there is no stark “to-survey-or-not” choice facing researchers.  Rather, there are many ways to combine data from diverse sources and capture the synergies among them.  For example, if you have a customer database it’s now quite easy to draw a sample of your customers and interview them.  In this way the “hard” and the “soft” are integrated into one data file. 

Taking this idea a step further, you can also incorporate your customer sample into a Usage & Attitudes study conducted among a more general population.  Respondents would thus include your customers and users of competitor brands.  You will not have access to competitors’ customer records, but with modern statistical methods it’s possible to impute missing data for competitors’ customers (though this should be done with care). 

Survey data, such as advertising and brand awareness and brand image, can be tracked over time and integrated with sales and other marketing data from various sources, including Social media.  Simple correlations and graphics can uncover important patterns among these variables.  Sophisticated market response/marketing mix modeling, in addition, can be used to assess marketing activity in detail, including competitors’, and improve marketing at both the strategic and tactical level.  

Another approach, still quite new, is tracking  Social Media activity (with permission) of respondents who have also participated in a consumer survey.  One application is in advertising research.


While the foregoing are not fresh innovations, they probably are not utilized to their fullest.  This brief overview has highlighted quantitative consumer surveys but it also seems doubtful that synergies among qualitative research and the New Data have been fully explored. 

American writer Mark Twain once famously declared that the report of his death was an exaggeration.  Survey research also has died before but remains alive and well.  New data sources will not make it irrelevant – they will make it more valuable.

Please share...

18 responses to “Why Survey?

  1. Great post, Kevin! Very relevant and timely…”Insights” are never one-dimensional, as you elegantly articulated on your post. At the risk of being overly simplistic, I liken it to driving a car – one must look ahead through the windshield (and to the sides), as well as use the rear view mirrors, to truly understand where the car (consumer) is going. Each provides a different and valuable vantage point and, to use one without the others is risky.

    Thanks again for putting it so well.

  2. Thanks Kevin. In all the discourse on how the vast amounts of social media data are going to replace traditional research no is asking the key question, what value do marketing management place on this data? The current discussions are all coming from market research suppliers and industry journalists who thing they are on the verge of some breakthrough in technology and information.

    I would argue the buyer is side is a lot more skeptical. It is clear that you can use findings to do the sorts of things that Amazon do – make offers to you when you buy a certain book category. But this kind of data only really applies to a captive data base. For most marketers the kinds of data they are now getting from social media have a lot of basic concerns around validity, representativeness and reliability.

    The data from feeds, blogs, forums and social media sites like Facebook and Twitter generally have no underlying structure. They are just data flows. Even if you have a database of customers providing regular communications just who are they and how representative are they of anything? Just a simple analysis will suggest marketer caution around huge age biases, gender biases, inexperience biases, complainer biases, heavy user biases, etc with the majority of that social media flow. Even the players in this consumer field don’t trust the information. Recent studies yet again confirm the growing distrust among social media users about online communications versus old forms of media. It make sense. Every customer service department knows you are not dealing with a representative sample of customers in that database.

    When we start to look at all this information from a marketing director’s perspective all I see is a fairly healthy degree of caution. Most social media data is seen as “nice to know, but how useful”. Sure there are insights out there and an occasional great story, but anyone who has worked in a major brand organization will know how hard it is to convince any senior management that all that sentiment analysis is nothing more than just unstructured word of mouth. The role played by traditional research will always be to provide that valid and reliable feedback to management. In-house departments will soon discover how easy it is to run the kinds of sentiment analysis stuff that some new-agers think is going to provide breakthrough brand strategies. I expect very little of of the internal market research dollars will directed to the hocus pocus around Twitter feed, Facebook sentiments or forum feedback. Companies will soon learn that everyone has the ability to run “big data” insights and ultimately they will all have the same information and same manipulative tools, if we can call Amazon’s offers as such. At the end of the day a good old fashioned MR will shine through even if it is captured by SoLoMo technology!

    1. I’m not sure who you are talking to @Chris, but just a little over a week ago over 100 client-side decision makers from some of the biggest brands in the world converged on Philly to engage very deeply indeed with these “New Age” suppliers, and in the case of 6 of those brands they met in private with about 80 of these suppliers and I know for a fact that many pilots and full on projects resulted. I also know of about $100M in tracking business that is being switched from survey to social media based methods in 2014 after careful side-by-side testing going on right now.

      I agree that traditional MR will always have a place and that augmentation is the primary model in place tight now, but to think that clients are not spending on other approaches is just plain wrong. Every industry barometer we have indicates a decline is MR spend globally; that doesn’t mean client insights budgets are being reduced (although in some cases that is certainly true), it means they are being shifted to vendors outside of what we consider to be MR and again, every barometer available validates that too.

      I am all for an honest dialogue about efficacy and fit for purpose, but whistling past the graveyard isn’t going to get us anywhere.

  3. I just read this today and thought it kind of summed up the marketing managers view on social media data. A poem by Yeats referenced the great Prince Guidalberto of Urbino who apparently decided to build his beautiful city without sending “runners to and fro/that he might learn the shepherd’s will”. There are a lot of shepherds out there!

  4. Leonard, whenever there are new technologies that hit the industry we see the kinds of response that you and your journalist connections talk up – the demise of the industry as we know it, disruption, get with the program or be left behind. It is no wonder that industry barometers confirm this fear. I doubt the revenue numbers, when they truly come in, free of the current recession globally, will confirm what you think is reality. Surveys of marketer opinions and market research players in this current environment are almost certainly misleading. The threat of the new always results in this kind of yea saying. The fact that you think this indicates the future is risible.

    Reality is otherwise as time will tell. Using phrases like “whistling past the graveyard” again show your biased orientation to instill this fear in the industry, because lets face it, it’s more interesting journalism than challenging the new age bull. No one I know at senior level in this industry is either unable to understand current trends nor are they at all concerned about all the developments on the social front because of two reasons. One it is very simple to get into the data collection and analysis of social media. Barriers to entry are negligible. We have written our own multi-Asian language search tool and were using it more than 3 years ago to manage complex key word analysis. Hadoop is available to anyone. There are more and more software out there that will enable this whole process.

    Secondly our clients are so far unimpressed with what comes back from social media. Lets face it, it has no structure, no representation, is potentially misleading and frankly falls in the category of customer service feedback where bias is a given and negativity assured. As I have said before, those who post complain the most. Coke, P&G, Nestle, etc are all reporting disappointing results from social media communication. For example any communications strategy on the mobile network is proving a dud. On top of that even social media users are very blunt in their perceptions about the trustworthiness and validity of feedback from this social millieu. Coke recently dumped on social media buzz as worthless for brand development. Even social user clearly trust mainstream media much more than Facebook or Twitter or Forums and Blogs. So where does this leave your massive information base that is supposed to change the history of marketing? I would say close to in tatters at the moment. Leaving out companies with bespoke panels like Amazon I would say most social media research is proving disappointing.

    On top of that I seriously doubt you journalists really understand the social media user. There is a mistaken belief that people are falling all over themselves to provide insights and feedback and creative solutions. Your whole premise is that people want to participate. And even worse they will do it on smartphones, that most clunky communications tool. Marketers have long known that the vast majority of brand users are not seeking one-to-one relationships, treat brands as very minor moments in their lives and frankly do not have a good creative idea in their brains. The problem is you fail to understand you are dealing with some segments of social media usage that exist in the long tail and do not represent the silent or indeed most of the majority in the least.

    The main point you seem to miss is that the whole underlying tenet of social media communications is you choose who you talk to, when and where, whereas marketing’s take is still I will be in your face one way or the other from push surveys to online ads. I think the disconnect is pretty important and something you need to reflect on.

    All I would suggest is that industry writers need to show a little less gushing support for the “wow, you better adjust or you will be left behind nonsense”. So, that honest dialogue issue you raise is something you may want to address.

    1. Ah @Chris, we should really organize a debate my friend; it would be fun.

      A point of clarification; I am NOT a journalist; I am a business person, research practitioner, analyst and consultant who happens to also have a blogging and editorial role. My opinions are formed by my business interactions, similar to yours. It sounds like we just have different client bases so we’re both maybe suffering from confirmation bias, or perhaps birds of a feather do flock together. Regardless, I have no agenda other than sharing what I am seeing and experiencing and my take on what it means.

      As to your points regarding the state of social media analytics as a marketing tool or insight generator the first generation of social solutions were cheap and flawed in many ways, and we tried to shove old models into new boxes and that is where the disappointment lies. Marketers, technology providers, researchers and consumers are recognizing that the value in this new era of massive behavioral data (that is really all social media is) is based on different assumptions than the old marketing metrics and those companies are seeing incredibly promising results. Your throwing the baby out with bathwater is risible as well. Attitudinal data, regardless of whether it is asked or inferred, is always suspect and only a small part of the puzzle. Our converged technology world goes much further with actual behavioral data through a million interactions across 100 million informational systems and THAT is where the value is being delivered. Why ask what someone is going to do when we can observe it and through those observations begin to predict it? That is absolutely happening today, and it’s only going to get better as we move forward.

      Let’s agree to disagree for the moment and continue to share our thinking openly; time will tell who is closer to the truth and I suspect it will be different from what either one of us thinks.

  5. I think Lenny and Chris are arguing the different sides of a mountain, Lenny going up while Chris is coming down. We saw the same dip in survey-based research when scanner data became popular that we’re seeing now with other non-obtrusive or pseudo-obtrusive measures. Kevin is right (IMHO) in that there are still places where survey will do a fine job, Chris is right in that some of this shall pass too, and Lenny is right that when large manufacturers are playing with new data, we shouldn’t completely rule out the possibility of there being something there.

    For me, what’s missing in this discussion is the need to test results of any analysis, whether it is survey based or “big data” based. Any model we create from any of this data (and an analysis is always a model, even if not a mathematical one) is only a model and does not necessarily predict truth. I’m a big fan of experimentation, testing whether an implied solution from any of the techniques talked about above actually work.

    1. Good points Steve. There is a comment on the discussion thread for Larry’s post on The Coming Rebirth of MR from Tony Cosentino that plays right onto this. Tony says it better than I do so I’m reposting it here as well:

      As a TNS alum and now a software industry analyst (I cover business analytics and big data, not survey software), I agree with many of the points in this well written post.

      I am curious how the author sees larger market research firms addressing these disruptive trends given what seems to me to be an entrenched interest in the old paradigm of data collection. Where do the big firms place their people, technology, process, and information bets in this brave new world? I’d be particularly interested in answers that are outside of data collection software since it seems SurveyMonkey, Google and others already have first mover advantage here.

      To add context, I reference the following quote and my own observations below it:

      “We can approach this from a “soft integration” type perspective, where we line up different sources of data and “triangulate” in on an answer. Or, we can approach it more from a “hard integration” type perspective and model and derive predictions using very different kinds (and volumes) of data. The latter is the province of the now famous (but rare bird indeed) “data scientist”.”

      I agree, but which direction will the MR industry head? There is a lot going on here- new visualization tools (e.g. Tableau) and analytic workflow tools (e.g. Alteryx) can help with data blending in the “soft integration” scenario and consulting firms are already picking up this low hanging fruit. The second area or what the author calls “hard integration” is where the revolution is really just getting started. Last week I attended the Hadoop Summit in San Jose where companies are starting to move past the data scientist bottleneck. What’s interesting is that technologies such as Hadoop allow for a massive storage and processing facility for raw un-modeled data. We can start prepping and analyzing data (all types including social sentiment data.) together on one integrated platform. While this revolution is still in its early stages, companies like SPSS, SAS, and Revolution Analytics (think industrial R language) are driving analytics onto this enterprise scale platform. With this, data exploration and analytics shifts into an entirely new world. Btw, the NSA platform that has received all the attention was built in this way. Privacy arguments aside, I recently spoke with one of the guys that built the NSA system and it’s really interesting the level of insight you can get when you are able to marry so many data sources!)

      …..So where do the larger firms go? Again, people and technology are top of mind to me, but process and information approaches would also be of interest!

      Thanks, and regards,

  6. Thanks to all for your perceptive and thoughtful comments. The extent and pace of change in the mr industry is breathtaking and I think the landscape will look very different in 10 years time. Not entirely different, though, and I agree that many tried-and-true tools will still have their place. Maintaining quality standards in the industry will increasingly prove challenging, but perhaps that is best treated as a separate topic. Thanks again!

  7. Kevin, nice article. Survey research at its best is about the future and underlying constructs. This is something data mining (Ooops, Big Data) is not built for. we can creat models to play what-ifs, but they don’t have data on future events and factors not present in the market or client’s offer. Surveys can at least incoporate possible worlds and measure those not in the client’s net.

    Social media, can best be summed as ‘if a million monkeys tweeted for a million years they would be tweeting what they tweeted a million years ago’. Like call centre data it shows the negative and often – ironically – the infrequent.

    While Big Data and social media have a lamp post we will find people under them looking for their insights. Yes, you will find some. But it is in the darkeness we will find more.

  8. Kevin, great points in the article. I think there will always be a time and place for survey-based feedback research because the resulting structured data tells a much better story than unstructured data from social media. Until the day comes when we can better associate the demographic profiles of social media users and their individual conversations, surveys will reign as the preferred methodology for insight generation. Sure, text analytics and sentiment analysis can tell the general tone and thematic composition of how a consumer feels towards one brand versus another. But it’s much more powerful to act like a journalist and probe 1,000 consumers with direct questions about the brands they buy – it just tells a better story in the end. Many say that consumer panels have become commoditized, biased, and unreliable – but I say that the survey research we generate from panel respondents can provide a much higher return on investment than social media if there are sound research methodologies driving the design and analytics from the resulting data and insights.

  9. I have enjoyed reading all of your views…..clearly we are in a transformational time. Having spend the last 10+ years testing “what if” scenarios for our clients using our virtual shopping platform recruiting “real” shoppers / consumers we have been extremely successful predicting outcomes I firmly believe measuring behavior is far more predictive than collecting stated opinions and attitudes. We continue to stretch our models to expand our understanding of the “why” behind decision making. We can layer on ethnography, biometrics, neuroscience, emotional measurements to enhance the story. When your business and research objectives are to move the needle, increase sales, get activation and implementation, increase brand equity, delight the Shopper than behavioral metrics are a must.

  10. As long as there is demand and supply and the customers have an opinion about it, the surveys cannot die. I have started using online survey tool – SoGoSurvey since a couple of years now and it really helps.

  11. Starting with the methodology seems to me to be to looking from the wrong end of the telescope. It depends firstly on what the business is trying to do – improve quality, increase profitability, monitor marketing effectiveness, design new products, uncover new customer needs… Once you have a sight of what you’re trying to achieve then pick and choose the data sources and research approaches. Plural – because no-one will be relying on one data source. A CSat survey will be combined with real production measures, customer service feedback and social media monitoring to understand quality and targets for improvement. A design team may not need a rep sample to help flesh out initial design proposals, but someone trying to predict market share will, and they’ll want to cross-analyse with sales. Social media fills a hole but you have to be so careful with representativeness and coverage – so it’s great for picking out general sentiments and, from the detail (not aggregate), can be a source of inspired thinking, but it’s just another piece for a researcher to use in the right situations.

  12. I agree with you Saul, but the article addressed another issue, namely the frequently-heard claim that survey research is dead. This claim, a very old one, has never withstood scrutiny.

  13. I agree with you Saul, but the article addressed another issue, namely the frequently-heard claim that survey research is dead. This claim, a very old one, has never withstood scrutiny. I know of no one these days who feels that surveys are the ONLY methodology, and, offhand, can’t think of anyone I know who ever did. 🙂

  14. Was running into CAPTCHA code errors for some reason, so will try again I should add that I would encourage you to write a post on the themes you mention, as they point out historical flaws in the way many of us have gone about our business in MR. The focus on methodology rather than business objectives, if anything, seems to be worsening.

Join the conversation