PureSpectrum - Schedule A Demo
Our new GreenBook Directory site is live!
COVID-19 guidance, tips, analysis - access full coverage here

Garbage In Garbage Out Part Deux (aka Panels Suck)

As a client-side researcher. And while I have a reasonable degree of confidence about what the sources of error and bias are in my CRM-based sampling efforts, my trust in panel recruiting erodes more and more every month.


Editor’s Note: All I can say here is don’t shoot the messenger folks! There is a reason that the the two aspects of MR that are facing massive new competitive threats are sample access and survey based research, and Jason explores a part of that reason in this post.


By Jason Anderson

In my role as a client-side researcher, I am blessed with an unbelievably large global customer database in the tens of millions of people. Not email addresses, but real people that I know exist in the real world by virtue of their purchase histories, credit card information, and behavioral metrics on our various online services. Because of this blessing, over 90% of the survey work I field is CRM-driven.

But there’s still that other 10%. And while I have a reasonable degree of confidence about what the sources of error and bias are in my CRM-based sampling efforts, my trust in panel recruiting erodes more and more every month. Consider, for example, a recent vendor selection experience:

  1. I decide to run a study in Country A, Country B, and Country C. I request bids from Company X, Company Y, and Company Z.
  2. I receive bids for $25 per complete, $20 per complete, and $6 per complete.
  3. I contact the local office for Company X in Country A and get an additional quote of $12 per complete.
  4. I tell Company Y that their $25 per complete bid has been beaten by a significant margin and get a “new” bid at $15 per complete.

Names obscured to protect the innocent. But nobody was “innocent” in this exchange, because from a distance it becomes obvious that the “value” of a completed survey is completely arbitrary and driven not by data quality or service quality but by a desire to win the bid.

Worse yet, I question whether that completed survey is even worth $6 to begin with. As an experiment, I joined one of the name-brand panels as a “panelist” under a pseudonym one month ago. I completed as little of the registration process as necessary to become qualified to take surveys. (Don’t worry, I haven’t polluted any of your actual work with fake responses. But I’ll come back to that in a moment.) Between August 3 and September 10, I received 25 survey invitations. That’s roughly 5 surveys per week. The panel’s frequency of contact guidelines explicitly say no more than one invitation every two days and no more than 12 per month.

“Oh, but it can’t be that bad! Most panelists are legitimate.” Let’s assume for a moment that this hypothesis is correct, and that panelists are recruited through completely legitimate efforts. For example, perhaps they were on Google and searched for “surveys” (see right).

Hmm. (And by the way: I’ve never been offered $20 to complete a panel survey. Which panel do I need to join?)

Creating a fake panelist account is fast and painless. Identity verification on the Internet is near impossible. But it’s not just respondents committing fraud in this process, the panels themselves are complicit. The lack of transparency in downstream processes invites opportunism, if not straight-up breaking of rules.

Consider: What’s the difference between a $6 respondent and a $25 respondent? Is the $25 respondent substantially better in terms of recruiting practices, data quality, and policy integrity? Or was the $25 respondent simply a $6 respondent that had been purchased from another source and marked up? And how can you tell the difference?

Answer: You can only tell the difference in quality if you are told which panels are being used, and how those panels manage their database. I haven’t found full-service research agencies to be terribly eager to share that sort of information, because:

  1. it allows me (with a little bit of sleuthing) to determine the profit margin between the sample cost and the delivered work, and
  2. the agency harbors fears of being disintermediated (there’s that word again!).

So what’s a client to do? I have three rules for myself:

  1. Always know the source of the sample.
  2. Never communicate “statistical margin of error” to internal clients on panel-based surveys.
  3. Stay close to the source. Don’t tolerate unreasonable mark-up on panel data, particularly when it’s known to be a pass-through cost from a downstream supplier.

Please share...

7 responses to “Garbage In Garbage Out Part Deux (aka Panels Suck)

  1. Your post definitely hit a nerve. “Panels” are one of the dirty little secrets that no one wants to talk about – and I talking about both vendors and clients.

    REPRESENTATIVENESS -“WHO ARE THOSE GUYS”? The whole reason we ‘sample’ in research, is because it would be infeasible to survey everyone ( the ‘population’). A critical requirement is that we need to be able to use our sample to make inferences about the entire popluation. The challenge is, who are these panelists really? Sure, they are real people – but, are they really who they say they are? If not, then you really can’t make inferences about the population – and so the research becomes worthless. Matter of fact, it is worse that that….you can actually get the WRONG answer. Online research is fast and cheap…but too few people really stop to ask whether the panelists really represent the true population.

    FALSE SENSE OF SECURITY Too many people assume that if I use Panel A for wave 1, and use Panel A for wave 2, that they are getting a consistent set of respondents, and trend is protected. This is often a false and dangerous assumption. Many panels turnover constantly – with the life of a panelist measured in months, not years. This means, a year later, it is not uncommon that the entire panel has been refreshed. Panel providers are in a constant mode of acquiring new panelists to replace panel mortality.

    YOU GET WHAT YOU PAY FOR The price of panelists varies greatly – but demand has unfortunately created a race to the bottom. I do mostly B2B. For a short period, I stood on my soapbox and evangelized using “high quality online panel sample – that was either phone recurited or phone verified.” I stressed, “you are making milllion dollar business decisions here – now is not the time to save $20 per complete. You want the truth, right? If yes, use the best quality panel sample you can afford.” For a while, some of my clients listened – and they were willing to pay a premium (1.5x to 2x) for better quality panel sample. But over time, I started losing more and more bids. I then offered an a la carte choice in my proposals (a) junk sample (b) so-so sample and (c) high quality sample. What happened was my clients started to vote with their wallets – choosing (a) and (b). So naturally, the panel providers stopped investing in (c)…because there was no demand.

    NEXT STEPS? I believe this is a serious, serious problem in our industry. It’s a dirty little secret that has been ignored far too long. But to fix it will require vendors, clients and panel providers working together to (a) agree on some definitions/requirements for quality and (b) create both demand and supply of quality sample. Until that happens, nothing will change – and you’ll have garbage in, garbage out.

    Thanks for raising this important topic!

  2. Far be it from me to defend the online panel industry. Like most other things, it’s caveat emptor. So you need to be an informed buyer.

    The ESOMAR 28 Questions to Help Research Buyers of Online Samples is an excellent tool to use in evaluating potential suppliers. Ignore the answers panel companies give on their websites and get the answers first hand. If you can find a panel company that has certified to ISO 26362 you can ask to see the specifics of pretty much everything they do.

    If all of that is more than you want to sign up for then choose an agency that you know you can trust to work with. If they do a high volume of online work they should already have vetted suppliers and sorted out most of the bad guys.

    There is another aspect to this story relative to clients and DIY but I’ll pass on that for now.

  3. Sorry, but I have some problems with the reasoning here. I’ve been a critic of the panel industry for years (Grey Matter’s two Dirty Little Secrets of Panel Research reports should demonstrate that), but the reasoning that widely divergent bids must mean a problem in quality just doesn’t hold water. I’ve gotten wide spreads in bids on panel sample, phone field work, data entry, focus group recruiting, mobile MR, and every other method throughout the years.

    Individual vendors may mark up their services differently, may understand the needs differently (or misunderstand them), may have different pricing structures, or may be more or less hungry for the work. I’ve frequently had vendors come back to me offering lower prices on qualitative recruiting or BTB phone work or whatever when I tell them they didn’t get the job. That has nothing whatsoever to do with the methodology – that’s the individual vendor.

    As for full-service vendors not revealing the sample source – again, that says nothing about the methodology, but it does say a whole lot about the vendors you’re choosing. I proactively discuss the sample source with clients, regardless of whether it’s panel sample or any other sample source. Ultimately, you have to use the data, so you have to feel comfortable with the approach I’m recommending. Rather than finding a new methodology, it sounds like you need to find some new vendors.

  4. The most important important thing in panel business is to engage the panelists. If a panel provider doesn’t getting enough business to engage the panelists then it usually happens to work on the bear minimum profit to get business flow for panelists engagement and saving the whole panel instead of charging premium and waiting for the business.

    The cost doesn’t necessarily associated with the quality of a panel.

  5. Answer: You can only tell the difference in quality if you are told which panels are being used, and how those panels manage their database.

    I disagree — the name, size or owner of the panel does not necessarily indicate the value or quality. The demand in this industry far outpaces the supply. As a result, our most important asset – the survey respondent – is quite often abused, misused and overused. Even the big players hammer their respondents to death. It’s a tough industry and it requires far more effort to manage the sample houses than ever before. If you want good survey results — make sure you know how to manage the sample houses. Find good project managers and stick with them.

    Lastly, everyone needs to understand we are all fishing from the same pond. Most respondents are on multiple panels. Forget the nonsense about “proprietary” panels — none of these companies own the respondent. They are fee to do as they please.

Join the conversation