Civicom: Your project success is our number one priority

5 Things the Sample Industry Must Do

The sample industry faces a number of challenges and must implement a number of changes to improve the quality of samples.

Editor’s Note: Sampling has always been a core concern of the survey research industry.  Without defensible sampling procedures, confidence in what we do will wither.  No wonder so much focus is placed on it, especially in light of all the technology-driven changes in how we do survey interviewing.  Brian Lamar gives us his POV on critical factors that the industry must do to improve the art and science. 


Recently at SampleCon, a conference dedicated towards improving marketing research sampling, there was a lot of discussion around the current challenges in online sampling, including how to improve the respondent experience, new gamification techniques, tips to stop fraud, and other trends in the industry.  During one of the last presentations, however, someone asked a question that struck me. He mentioned that he has been in market research for 40 years, and the issues that were being discussed were the same issues that had been discussed 10, 20, even 35 years ago. Unfortunately, our industry is stubborn and slow to get at the heart of the issue.  

Here are the 5 things that we must do as an industry to improve sampling:

1. Create a Respondent Global ID

It’s not a well-kept secret that many respondents are members of multiple online panels – we just don’t have a quantifiable number.  If all panel companies were to adopt a global ID for their panelist that they shared across the industry, it would allow for better results since we could account for panel membership overlap and de-duplication.

Yes, most panel companies employ some form of digital fingerprinting software to block the same device from taking multiple surveys, but with the proliferation of devices and the fact that most people have 2-4 now, what’s stopping someone from taking the same survey on different devices for different panels?

That’s why there needs to be a universal verification system as part of any respondent global ID solution.  We, as an industry, spend far too much time designing surveys to validate and verify respondents, as well as evaluating responses after the survey is submitted.  A universal verification system will give confidence to sample buyers that the data they are getting is from verified, unique respondents, as well as make us in the industry more efficient.

2. Implement a Rating System

In addition to the global ID and verification system, a standardized industry-rating system should be implemented.  This system would be two-fold, including a respondent rating component and a survey rating component to evaluate client surveys.  

With the respondent rating component, end users of the data provide an Uber-style rating of respondents.  While many Blockchain companies are developing this as part of their offering, why couldn’t this be implemented at the traditional panel level?  It could essentially be another targeting criteria for buyers, as a tool for panel companies to improve their respondents. While it may be overwhelming, it could initially simply be a binary rating: removed or not removed.

As a bonus, companies could publish the overall results so sample buyers can be more educated in their purchasing.

For the survey rating component, this would utilize the same framework as the respondent rating component, but have respondents evaluate and rate client surveys.  If the survey is longer than anticipated, it will be reflected in the ratings. If the survey is boring or irrelevant, the same would happen. We could then create benchmarks and help educate those that score poorly on how to improve the survey experience or incentivize more.

3. Better Routing / Better Profiling

With almost any survey, there are questions asked that a respondent answers over and over again.  Whether it’s demographic or profiling questions, or the same questions on different surveys that are routed to them, these are a headache for respondents.  We can fix this by passing along data on a respondent, starting with demographic data, but expanding to other areas like behavioral and attitudinal profiling data.  This will shorten surveys and LOI and make a better survey experience for the respondent.

4. Change the Incentive Model

The current model with many panel companies is to only pay the incentive for people who complete the survey and their data is not deemed “poor quality.”  One of the most frustrating experiences for a respondent is that they invest a significant amount of time in a survey only to term out near the end, therefore receiving nothing.  This needs to change. We should be adequately compensating respondents, especially if they have been verified and willing to participate. I am not saying provide an equal incentive, but a menial incentive would go a long way.

This isn’t a new idea either.  There are some companies that do this, and they are creating more loyal respondents and diminishing churn in their panel.

5. Be Truly Transparent

There are times where panel companies are less then forthcoming about where their sample comes from.  Sometimes it is their proprietary panel, sometimes it’s from partners or even other sources, but they may not tell their client about the other avenues. They may even try to play it off as if it is all their proprietary sample even when it isn’t.  All companies need to be completely transparent. This includes detailing your recruitment methods, your response rates, and the partners (both internal and external) utilized on any specific study. We need an extra level of transparency to help buyers.  

Here is the problem: there is no pressure on our industry’s governing bodies to push for any of these changes.  CASE, a recent industry consortium of researchers from both the client and supply side, has stated they want more information when purchasing sample, and they want to have access to more information.  We need to demand that industry associations such as The Insights Association, ESOMAR, or others step up and issue strong guidance towards best practices.

I believe that if we, as an industry, make these changes, we’ll see:

  • Respondent experience improve greatly, leading to increased response rates and reduced turnover.
  • Membership of online panels will grow.  We don’t know the exact number of panelists in the industry, but we know it’s not enough.  Increasing panel membership will make panels more representative while reducing the overall burden.
  • Quality will improve and better business decisions can be made.  This is what will drive the change. I am a strong believer that our industry can grow exponentially over the next few years, if we increase the ability for brands to make better business decisions and measure those decisions.

Let’s make sure we don’t have the same conversation 35 years from now.

Please share...

2 responses to “5 Things the Sample Industry Must Do

  1. Great article Brian! Specific suggestions are what we need to organize and prioritize. Taking a strong position, publicly, in writing, requires courage and passion. Thank you!

  2. The problem is – what’s the incentive for panel companies to do any of this? Clients keep buying crappy sample and demanding that the CPI gets lower and lower. I’m amazed at how many questionnaires and projects I see that have no quality checks to them – the data is collected and accepted, with the fervent hope that every response is legit. Until clients demand change, give business only to panels that are trying to make positive improvements in quality, and stop the absurd price pressure, panel companies have no incentive at all to improve things.

Join the conversation

Brian Lamar

Brian Lamar

Vice President, Insights , EMI Research Solutions