PureSpectrum - Schedule A Demo
Our new GreenBook Directory site is live!
COVID-19 guidance, tips, analysis - access full coverage here

Market Research: It’s Time To Put Up Or Shut Up

The pace of technological change is breathtaking. It’s a fantastic time to be working in this space. We can, however, control our own actions. It is time for us to “walk the walk.” We need to follow the principles we espouse even if they are commercially inconvenient.

put up or shut up


By JD Deitch (@JDDeitch) 

The entire research industry just took a thunderous body blow.

This article in the New York Times explains the issue with minimal jargon and an implication that’s clear as a bell: there’s a big problem in polling and, by implication, in online research. It’s no longer an “internal discussion,” nor is it just a British problem. It’s mainstream news.

Do we all understand how serious this is? Do we under understand that the majority of people who read that article will conclude benignly (or self-servingly) that online polls are crap?

We’ve been wringing our hands for years and now the chickens have come home to roost. It’s time for the all of us—research buyers, research agencies, sample companies—to stop blaming each other and start taking responsibility for what is now an industry crisis.

This isn’t news

Bias and sampling issues are part and parcel of survey research. Properly trained researchers are indoctrinated in these issues and, if my own experience is any measure, spend their professional lifetimes trying to find ways of managing and mitigating these concerns. Steve Coffey, long-time innovator at the NPD Group and former Chairman of the ARF (and my mentor), had a saying that sticks with me to this day: you have to be a grownup to do survey research. There are real trade offs. Technology and behaviors change, sometimes dramatically, as they have over the past several years in the always-connected smartphone age.

There’s far greater diversity of methods … [that reflect] the still-developing science of Internet polling. No one is sure of the right way to do it.

Nate Cohn, “Online Polls Are Rising. And So Are Concerns About Their Results.” New York Times, November 27, 2005.

In the modern era, concerns about panel quality are hardly new. But they are now boiling over into the public domain. Yet as the most recent GRIT study indicates, the industry is still a long way off from being mobile ready despite YEARS of information and data and analysis and commentary calling for change.

Research buyers are slow to change. It’s hard work, with increasingly smaller budgets, and they are struggling to discern the signal from the noise among substitute and complementary methodologies. Traditional suppliers, whose fees are a direct function of length of interview and who count on annual revenue streams from inflexible tracking and normed studies, are still trying to stave off financial challenges (rather unsuccessfully) to buy time to retool. Panels are starved of respondents under the age of 35 because none of them wants to take long punishing desktop-only surveys, which as we’ve known for years cause them to disengage and respond with bogus data. These aren’t fly-by-night companies either. These are industry leaders. Meanwhile, the combination of new tools and perceptions (right or wrong) of the inadequacy of survey research is driving down willingness to spend.

It’s a terrible vicious circle that has all the elements of a prisoner’s dilemma.

It doesn’t have to be this way

Blame for the industry’s problems can be placed at every point in the chain. But it doesn’t have to be this way.

It is an indisputable truth that survey research can be done far more inexpensively, far more quickly, and far more efficiently with online platforms. These platforms are device agnostic and mobile friendly by design. They are automated and don’t require a slow costly army of people to run.

It is an indisputable truth that shorter, well written surveys still yield actionable information and meaningfully reduce “bad behavior” whether in the form of real deliberate fraud or the far more frequent inattentiveness for which researchers only have themselves to blame.

It is also an indisputable truth that people who aren’t subjected to long, poorly written desktop-only surveys are better responders who complete more surveys more accurately on average in their “lifetimes”.

It is also an indisputable truth that the technology exists to automatically test and retest people’s answers, and to use analytics and game mechanics to reward accurate responders, to not over-invite them, and to eliminate those over time who give suspicious answers, to treat panelists fairly and reward them for their rich profiles while maintaining their privacy.

Taking responsibility

There’s really no reason for the current situation to persist. It no longer requires management courage to reach the conclusion that the time to change is now. If anything, those who aren’t articulating concrete plans should be challenged by their employers, boards, and shareholders. So let’s do that.

Buyers, sample companies, research agencies, listen up.

Research buyers, if you’re still running long desktop-only studies, you are a fundamental cause of this problem. Blaming your suppliers for the quality of their panelists is like blaming the bartender for your hangover. I get that the change is difficult, but unless you really don’t care about people under 35 or moms with kids or ethnic minorities, you’re increasingly buying junk. This has to be part of the 2016 plan.

Sample companies, really it’s time to stop. The education you’ve done on the conference circuit is important, as are your tools and metrics to help your agency or end clients adjust to the new normal. But if you’re still taking the work that’s killing your panels then you are part of this problem.

It is even greater hypocrisy to engage in the gratuitous theater of parading around live respondents who talk about cruising mindlessly through surveys and even deliberately lying in qualifying questions when you don’t even follow your own guidelines about when to let in mobile respondents. You know who you are. You know what the solution is to this.

Research agencies, please actually continue doing what you’re doing. Your clients are starting to abandon ship in numbers, and they’re coming to me and companies like mine for a better and cheaper solution. Let’s be equally clear: we’re getting better data because people aren’t tuning out and we aren’t missing big chunks of the population. Your lunch is delicious. Thanks for sharing.

A Challenge to the Industry

There are many things happening in and around the research industry that are out of anyone’s control. The pace of technological change is breathtaking. New sources of data and new methodologies are changing the way we think about the behavior factors that influence it. It’s a fantastic time to be working in this space.

We can, however, control our own actions. It is time for us collectively to “walk the walk.” We need to follow the principles we espouse even if they are commercially inconvenient.

  • We will be device agnostic and optimized for mobile by design.
  • We won’t subject research participants, ours or anyone else’s, to long crappy desktop-only surveys.
  • We will talk to clients and research users in plain language about what surveys are good at and what they aren’t and how that impacts their studies.

Consider the gauntlet thrown down. Over to you, industry.

Please share...

5 responses to “Market Research: It’s Time To Put Up Or Shut Up

  1. Another clear message to the market research industry and a welcomed set of views. However, to take this challenge to those who pose the problem, this message needs to go beyond the medium of Greenbook, Linked In and social media. Let’s hope the New York Times has high readership amongst market research professionals!!

  2. Nice, hard-hitting overview, JD.

    The MR industry could do itself a service if the main MR (and polling) industry organizations around the globe joined forces with leading academics to develop simple, practical guidelines for questionnaire design. I’m thinking of actual templates, translated into the main languages, for different kinds of surveys. Some MR companies and software vendors have done this for their own purposes, but making available something concrete that anyone conducting survey research could refer to would help, IMO.

  3. The foundation of market research is on the sample. If cheap panels can’t deliver then researchers will go elsewhere I totally agree. But this seems to claim that we have unrepresentative samples is because we have bad questionnaires, so if we just had better questionnaire techology and shorter surveys it would all be OK? Sorry I don’t believe this.

    Yes, a bad questionnaire will leave people cold and unwilling to take part further (I was amazed to run into a survey for The Guardian of all people that didn’t work on an iPad just last week – duh?). But the gap we have now is partly about reaching the right groups, partly about willingness to partake, and partly about panels that have become skewed towards those who like taking surveys.

    Reaching the right groups gets harder. Teenagers are on Whatsapp and use email barely at all or maybe never. Older folks detach themselves from the internet, or use it for short periods. Mobile numbers continue to explode with more than one number per person and fewer fixed lines. Ad blockers are signs that individuals are opting out of marketing land all together. If you block ads would you really be a keen participant for research? And those who regularly take surveys learn from the process and slowly their answers diverge from the population they’re supposed to represent.

    And this is to be solved wiht a better, shorter questionnaire? In fact some of the worst questionnaires I’ve seen recently have been very short, so length is no indicaction of quality.

    it’s back to fieldwork and sample design. A researcher has to think about the who and how before the what.

  4. Not every survey can achieve what it needs to do on mobile only platform. Some surveys still need to be longer and completed on non-mobile device

    And a focused approach to cleaning out low quality respondents can be effective.

  5. It’s been many years, but I still have vivid memories of having had to explain to a client that you can’t make respondents in a mall intercept survey “representative” by weighting by age and sex. I had to be very tactful since the client evidently was being deceived by a supplier they trusted.

Leave a Reply to colin white Cancel reply