By JD Deitch (@JDDeitch)
The entire research industry just took a thunderous body blow.
This article in the New York Times explains the issue with minimal jargon and an implication that’s clear as a bell: there’s a big problem in polling and, by implication, in online research. It’s no longer an “internal discussion,” nor is it just a British problem. It’s mainstream news.
Do we all understand how serious this is? Do we under understand that the majority of people who read that article will conclude benignly (or self-servingly) that online polls are crap?
We’ve been wringing our hands for years and now the chickens have come home to roost. It’s time for the all of us—research buyers, research agencies, sample companies—to stop blaming each other and start taking responsibility for what is now an industry crisis.
This isn’t news
Bias and sampling issues are part and parcel of survey research. Properly trained researchers are indoctrinated in these issues and, if my own experience is any measure, spend their professional lifetimes trying to find ways of managing and mitigating these concerns. Steve Coffey, long-time innovator at the NPD Group and former Chairman of the ARF (and my mentor), had a saying that sticks with me to this day: you have to be a grownup to do survey research. There are real trade offs. Technology and behaviors change, sometimes dramatically, as they have over the past several years in the always-connected smartphone age.
There’s far greater diversity of methods … [that reflect] the still-developing science of Internet polling. No one is sure of the right way to do it.
Nate Cohn, “Online Polls Are Rising. And So Are Concerns About Their Results.” New York Times, November 27, 2005. http://www.nytimes.com/2015/11/28/upshot/online-polls-are-rising-so-are-concerns-about-their-results.html
In the modern era, concerns about panel quality are hardly new. But they are now boiling over into the public domain. Yet as the most recent GRIT study indicates, the industry is still a long way off from being mobile ready despite YEARS of information and data and analysis and commentary calling for change.
Research buyers are slow to change. It’s hard work, with increasingly smaller budgets, and they are struggling to discern the signal from the noise among substitute and complementary methodologies. Traditional suppliers, whose fees are a direct function of length of interview and who count on annual revenue streams from inflexible tracking and normed studies, are still trying to stave off financial challenges (rather unsuccessfully) to buy time to retool. Panels are starved of respondents under the age of 35 because none of them wants to take long punishing desktop-only surveys, which as we’ve known for years cause them to disengage and respond with bogus data. These aren’t fly-by-night companies either. These are industry leaders. Meanwhile, the combination of new tools and perceptions (right or wrong) of the inadequacy of survey research is driving down willingness to spend.
It’s a terrible vicious circle that has all the elements of a prisoner’s dilemma.
It doesn’t have to be this way
Blame for the industry’s problems can be placed at every point in the chain. But it doesn’t have to be this way.
It is an indisputable truth that survey research can be done far more inexpensively, far more quickly, and far more efficiently with online platforms. These platforms are device agnostic and mobile friendly by design. They are automated and don’t require a slow costly army of people to run.
It is an indisputable truth that shorter, well written surveys still yield actionable information and meaningfully reduce “bad behavior” whether in the form of real deliberate fraud or the far more frequent inattentiveness for which researchers only have themselves to blame.
It is also an indisputable truth that people who aren’t subjected to long, poorly written desktop-only surveys are better responders who complete more surveys more accurately on average in their “lifetimes”.
It is also an indisputable truth that the technology exists to automatically test and retest people’s answers, and to use analytics and game mechanics to reward accurate responders, to not over-invite them, and to eliminate those over time who give suspicious answers, to treat panelists fairly and reward them for their rich profiles while maintaining their privacy.
There’s really no reason for the current situation to persist. It no longer requires management courage to reach the conclusion that the time to change is now. If anything, those who aren’t articulating concrete plans should be challenged by their employers, boards, and shareholders. So let’s do that.
Buyers, sample companies, research agencies, listen up.
Research buyers, if you’re still running long desktop-only studies, you are a fundamental cause of this problem. Blaming your suppliers for the quality of their panelists is like blaming the bartender for your hangover. I get that the change is difficult, but unless you really don’t care about people under 35 or moms with kids or ethnic minorities, you’re increasingly buying junk. This has to be part of the 2016 plan.
Sample companies, really it’s time to stop. The education you’ve done on the conference circuit is important, as are your tools and metrics to help your agency or end clients adjust to the new normal. But if you’re still taking the work that’s killing your panels then you are part of this problem.
It is even greater hypocrisy to engage in the gratuitous theater of parading around live respondents who talk about cruising mindlessly through surveys and even deliberately lying in qualifying questions when you don’t even follow your own guidelines about when to let in mobile respondents. You know who you are. You know what the solution is to this.
Research agencies, please actually continue doing what you’re doing. Your clients are starting to abandon ship in numbers, and they’re coming to me and companies like mine for a better and cheaper solution. Let’s be equally clear: we’re getting better data because people aren’t tuning out and we aren’t missing big chunks of the population. Your lunch is delicious. Thanks for sharing.
A Challenge to the Industry
There are many things happening in and around the research industry that are out of anyone’s control. The pace of technological change is breathtaking. New sources of data and new methodologies are changing the way we think about the behavior factors that influence it. It’s a fantastic time to be working in this space.
We can, however, control our own actions. It is time for us collectively to “walk the walk.” We need to follow the principles we espouse even if they are commercially inconvenient.
- We will be device agnostic and optimized for mobile by design.
- We won’t subject research participants, ours or anyone else’s, to long crappy desktop-only surveys.
- We will talk to clients and research users in plain language about what surveys are good at and what they aren’t and how that impacts their studies.
Consider the gauntlet thrown down. Over to you, industry.