Unearth the power of declared data
Insights That Work

The Dangers Of Assuming The Status Quo Is Right

One benefit of spending years in the market research industry is seeing the same arguments come round every so often. The first time one sees a particular argument it seems novel and convincing. But by the second or third time, one can better put it all in context.

Editor’s Note: In his usual erudite fashion, Ray Poynter challenges us to question new techniques and approaches but also to do the same to our own assumptions about what is the “right way” to do things in MR. It’s a great reminder for us all to be mindful of our own biases and preconceptions regardless of where you are on the “new vs. traditional” research spectrum. We should let the data lead us to conclusions, not our preconceptions, right?

By Ray Poynter

One of the benefits of having spent over thirty years in the market research industry is seeing the same arguments come round every so often. The first time one sees a particular argument it seems novel and convincing, but by the second or third time round one starts to becomes accustomed to it and to be better able to put it in context.

When telephone interviewing (often referred to as CATI – computer aided telephone interviewing) came on the scene in the 1980s people were concerned about whether it would introduce biases due to sampling issues and/or modality effects. These were valid concerns, but they were usually addressed by running comparative studies with face-to-face studies, as if face-to-face were a reliable indicator of ‘truth’.

Instead of asking ‘Is CATI the best available system?’ or ‘Is CATI good enough?’ or perhaps ‘What is CATI good for?’ the typical question that was asked was one almost guaranteed to make CATI look like a second-rate compromise, adopted because it was cheaper and faster. That badly formed question was ‘Is CATI biased?’. Yes, of course it was, everything that involves people measuring people is biased (as I explore in a recent VCU paper titled ‘Are Community Panels Biased?’). If research is to move ahead new techniques should be held to the same standard of inquiry and evaluation as existing techniques. They should not be held to some higher standard, predicated on an implicit assumption that what is already happening is perfect and that new techniques need to produce the same results as a necessary minimum standard of acceptability.

When online research appeared, when online access panels appeared, and when research communities came on the scene, the same poorly constructed questions were asked ‘Is it fully representative?’ and ‘Is it biased?’, as if the other research alternatives were somehow beyond question.

In the last few years researchers have become increasingly aware that our status quo assumptions of rational respondents, being reached representatively, and answering in an informed way are often wide of the mark. Neuroscience, behavioral economics, theories of social behavior, and predictive markets have all shown that market researchers need to re-address the status quo as well as properly addressing emerging ideas.

Research is on the edge of a revolution in the way it conducts its business. Key changes include: research communities, mobile devices, social media, and Big Data. Research needs to work out what these new strands can add to the mix, rather than trying to fit them to some hypothetical and outdated model of what research is and how it should be evaluated.

Key questions for research to address are:

      1. What approaches work, for what situations, and for which contexts?
      2. For any specific approach, what it strengths and its limitations?
      3. For a given business decision, what is the right trade-off between accuracy, speed, money, and depth? (We used to talk about quality, speed, and money – but with many research techniques the quality issue needs to be subdivided into accuracy and depth.)
      4. How do we ensure that research buyers are in a position to make informed choices between the various options?

Please share...

8 responses to “The Dangers Of Assuming The Status Quo Is Right

  1. I know this will shock you, Ray, but I’m going to disagree with you. Not about your discussion of evaluation criteria or the four questions for research to address, but the premise that these new techniques represent a revolution. They are [relatively] new tools which may or may not play out as valuable, enabling us to ask new types of questions. But I’m not sure CATI, Split-Cable Markets, Scanner Data, and Internet interviewing were any less impactful on our industry than the new tools are or will be. Yet I’m not sure we would call all of this old stuff “revolutionary”. They were new when they came out, they allowed us to ask questions differently than the old technique, but they didn’t fundamentally change what we do as researchers.

    And in the end, I think the way you evaluate a new research tool is whether it does a good job of mapping reality; are the decisions you make with the new tool better than the ones you make with the old tool, or if the same, are they faster and cheaper.

  2. Just re-affirms the 2 main principles:

    1) There is no such thing as a fully representative random population sample – different methodologies are better and more appropriate for some samples than others

    2) There is no such thing as the ultimate truth when researching people, BUT researching the right people, the right way, with the right questions is much better than no researching them

  3. I believe that the conversations held amongst researchers over the last forty years have been too focused on methodology. The conversations of clients – of luxury, prestige or premium brands – now are more about deeper understanding of the perceptions (as well as behaviour and motivations) of their customers (so they can engage them), better measurement techniques and metrics for improving marketing accountability/ROI. Too many marketing dollars are still wasted.

    Some clients now are looking to newer models as to how marketing, branding or advertising works. The best I believe are brand within similar culture specific (for uniquely positioned brands – think Apple, Coca Cola, Dove, BMW…) as opposed to industry standard with questionable norms.

    Researchers too should be having conversations on models before methodology.

  4. Let’s start by getting our terms straight. Jason likes a definition of “revolutionary” as new or innovative. That waters down some the definition I learned in high school when a “revolution” was a sudden, radical or complete change. So let’s stop talking about a revolution in MR. Things are changing, as they always do, but it’s hardly sudden and the penetration of new methods into total research spend is unremarkable thus far.

    Sure, there are lots of new tools and we should not hesitate to use them when they fit the client’s purpose. But that requires researchers as well as clients to understand the strengths and weaknesses of those tools and, thus far, I’ve seen little evidence that they do.

  5. One possible step in the suggested direction is to force ourselves to dedicate part of our time to learn about how the status quo has been disrupted by other colleagues, and to always include in our research proposals some innovative (alternative) approach.

Join the conversation