Editor’s Note: Sampling has always been a core concern of the survey research industry. Without defensible sampling procedures, confidence in what we do will wither. No wonder so much focus is placed on it, especially in light of all the technology-driven changes in how we do survey interviewing. Brian Lamar gives us his POV on critical factors that the industry must do to improve the art and science.
Recently at SampleCon, a conference dedicated towards improving marketing research sampling, there was a lot of discussion around the current challenges in online sampling, including how to improve the respondent experience, new gamification techniques, tips to stop fraud, and other trends in the industry. During one of the last presentations, however, someone asked a question that struck me. He mentioned that he has been in market research for 40 years, and the issues that were being discussed were the same issues that had been discussed 10, 20, even 35 years ago. Unfortunately, our industry is stubborn and slow to get at the heart of the issue.
Here are the 5 things that we must do as an industry to improve sampling:
1. Create a Respondent Global ID
It’s not a well-kept secret that many respondents are members of multiple online panels – we just don’t have a quantifiable number. If all panel companies were to adopt a global ID for their panelist that they shared across the industry, it would allow for better results since we could account for panel membership overlap and de-duplication.
Yes, most panel companies employ some form of digital fingerprinting software to block the same device from taking multiple surveys, but with the proliferation of devices and the fact that most people have 2-4 now, what’s stopping someone from taking the same survey on different devices for different panels?
That’s why there needs to be a universal verification system as part of any respondent global ID solution. We, as an industry, spend far too much time designing surveys to validate and verify respondents, as well as evaluating responses after the survey is submitted. A universal verification system will give confidence to sample buyers that the data they are getting is from verified, unique respondents, as well as make us in the industry more efficient.
2. Implement a Rating System
In addition to the global ID and verification system, a standardized industry-rating system should be implemented. This system would be two-fold, including a respondent rating component and a survey rating component to evaluate client surveys.
With the respondent rating component, end users of the data provide an Uber-style rating of respondents. While many Blockchain companies are developing this as part of their offering, why couldn’t this be implemented at the traditional panel level? It could essentially be another targeting criteria for buyers, as a tool for panel companies to improve their respondents. While it may be overwhelming, it could initially simply be a binary rating: removed or not removed.
As a bonus, companies could publish the overall results so sample buyers can be more educated in their purchasing.
For the survey rating component, this would utilize the same framework as the respondent rating component, but have respondents evaluate and rate client surveys. If the survey is longer than anticipated, it will be reflected in the ratings. If the survey is boring or irrelevant, the same would happen. We could then create benchmarks and help educate those that score poorly on how to improve the survey experience or incentivize more.
3. Better Routing / Better Profiling
With almost any survey, there are questions asked that a respondent answers over and over again. Whether it’s demographic or profiling questions, or the same questions on different surveys that are routed to them, these are a headache for respondents. We can fix this by passing along data on a respondent, starting with demographic data, but expanding to other areas like behavioral and attitudinal profiling data. This will shorten surveys and LOI and make a better survey experience for the respondent.
4. Change the Incentive Model
The current model with many panel companies is to only pay the incentive for people who complete the survey and their data is not deemed “poor quality.” One of the most frustrating experiences for a respondent is that they invest a significant amount of time in a survey only to term out near the end, therefore receiving nothing. This needs to change. We should be adequately compensating respondents, especially if they have been verified and willing to participate. I am not saying provide an equal incentive, but a menial incentive would go a long way.
This isn’t a new idea either. There are some companies that do this, and they are creating more loyal respondents and diminishing churn in their panel.
5. Be Truly Transparent
There are times where panel companies are less then forthcoming about where their sample comes from. Sometimes it is their proprietary panel, sometimes it’s from partners or even other sources, but they may not tell their client about the other avenues. They may even try to play it off as if it is all their proprietary sample even when it isn’t. All companies need to be completely transparent. This includes detailing your recruitment methods, your response rates, and the partners (both internal and external) utilized on any specific study. We need an extra level of transparency to help buyers.
Here is the problem: there is no pressure on our industry’s governing bodies to push for any of these changes. CASE, a recent industry consortium of researchers from both the client and supply side, has stated they want more information when purchasing sample, and they want to have access to more information. We need to demand that industry associations such as The Insights Association, ESOMAR, or others step up and issue strong guidance towards best practices.
I believe that if we, as an industry, make these changes, we’ll see:
- Respondent experience improve greatly, leading to increased response rates and reduced turnover.
- Membership of online panels will grow. We don’t know the exact number of panelists in the industry, but we know it’s not enough. Increasing panel membership will make panels more representative while reducing the overall burden.
- Quality will improve and better business decisions can be made. This is what will drive the change. I am a strong believer that our industry can grow exponentially over the next few years, if we increase the ability for brands to make better business decisions and measure those decisions.
Let’s make sure we don’t have the same conversation 35 years from now.