PureSpectrum - Schedule A Demo
Qualtrics: Here to Help

The Future of Panels:Five Investments Needed to Improve Participation and Reduce Bias

Among the many challenges market research firms face, none is more severe or important to solve than the mounting difficulty in securing a representative sample of survey research respondents.



By Jonathan D. Deitch, Ph.D., Global Head of Sampling, Ipsos

In the dark corners of the bar after a long day of conference presentations, back in prehistoric times before online panels but even as recently as pre-social media, conversations among researchers always turned to one subject: the disasters that had befallen each of us. As a former colleague once quipped, when you’ve worked in the business long enough, you know where all the bodies are buried.

Nowadays the conversations turn to a different subject: the trouble with panels. To be clear, nobody believes there’s a problem acquiring and retaining people who came of age before the Internet and time-shifted TV. The problem is everyone else. Scour the panel companies (or full-service shops with panels) for people under 35 or the parents with kids and you’ll find precious few. Racial and ethnic minorities, too. They aren’t joining traditional research panels any more.

The adoption en masse by these subpopulations of portable, always-connected devices is only a symptom of the industry’s pathology, one which device-agnostic surveys helps to alleviate.

The real disease, though, is that in general panels don’t offer compelling value to these people. The situation is critical in the U.S. and many countries in Western Europe (including the UK) and is becoming so elsewhere.

The impact of this crisis will be a real change in how sample is sourced, how much it costs to access real respondents, and on the future of panels themselves.


The problem is easily summarized. Research participants who have even a modest interest in rewards quickly figure out that, to earn enough points, they have to take way too many surveys, most of which aren’t compatible with their preferred device or the amount of time they’re prepared to dedicate for meager nonfungible compensation.

Those who feel more altruistic or simply want to speak their mind have plenty of direct outlets to engage with their favorite brands or vendors. These outlets often provide recognition in the form of return contact. Likewise, most Internet polls show results immediately. Users get the instant gratification of knowing if their answers are “right”, or at least how others have responded. The research industry rarely offers this. Studies are blinded for fear of self-selection bias. Results are hidden, the confidential property of our clients.

If the slow and steady decline of panel participation is an infection, the industry’s response has been to continue to apply leeches. Acquisition techniques are evolving, but panel management and retention practices have remained static. Engagement is still largely transactional. Competition grows fiercer by the day. Attrition is a big problem.

In fairness, panel companies are caught in a Bermuda Triangle of sorts. They face a sea change in respondent behavior driven by constant connectedness, one they are powerless to control. Their clients understand the need for change but many are unprepared or slow to accept it. Their managers (or shareholders) are concerned about financial risk given the high levels of uncertainty. Like a drug-resistant disease, no existing medications will help.

Building the Panel of the Future

Panels will decay into mere email lists (or social media “likes”) without new thinking and action around every aspect of the respondent lifecycle, from acquisition, to participation, to engagement and retention.

1: Reach and diversity are essential

Having fished the same waters for too long, panel companies have been forced to look to other sources to expand the top of the recruitment funnel. One of first was incentivized networks, or the “river”. Without a doubt, river/non-panel sources will become more important, which requires rebooting the conventional wisdom for many researchers.

Managed properly, the only real difference between a panel respondent and a river respondent is that the latter has decided not to join a rewards community. This is from whence we come methodologically, yet there are clients who consider river respondents to be of lower quality than panelists when, methodologically, the sources are equivalent: neither can be linked back to the population with a known probability of selection. It is ironic that these same people ask what suppliers are doing to combat “professional” panelists. Nobody has found the Goldilocks respondent yet, though we’re all looking. At least in the short term, and probably for longer, the river will continue to supply audiences that are increasingly shunning panel.

Firms will need to invest heavily in making programmatic acquisition work as well. The challenge in a programmatic world is not so much the right demographic profiles, but finding those willing to participate. Buying on impressions (CPM) is an easy way to lose a lot of money if the people one acquires don’t take surveys. The ability to recruit richly at scale and evolve as the market evolves will be an asset that provides competitive advantage.

Implied in this greater attention to reach and diversity is the need for new measures of quality. Panel size and response rate are becoming meaningless. Both can be artificially manipulated in ways that do little to really help us understand bias. Response rate simply isn’t applicable in a non-panel environment. What matters now is feasibility across an ever-evolving panoply of sources. Reach and diversity are the only ways to increase feasibility and reduce coverage bias.

2: Incentives and engagement must be meaningful

Reach and diversity help with the question of “who”, but not “why”. Panels are in desperate need of a new value proposition.

The easiest target for change is the proprietary incentive system, that is, those rewards that only have value through single panel participation. Common sense suggests these are of little interest to the audiences the industry needs the most. It seems safe to conclude that the level of participation required to earn a meaningful reward is too high. Companies that solve the incentive problem will allow respondents to accumulate points in multiple diverse ways and have fungible currencies. There are panel companies that do this already. Incentivized ad networks that form the backbone of many river sample offerings do this as well.

There are more interesting challenges on the incentives and engagement front, though. One is the impact that shorter, device-agnostic studies will have on retention. The burden on the panelist will be less onerous, but the incentive will be lower, too. This seems to still point to the need for transferable currencies.

Another is the possibility of differential incentives by respondent type. The only relationship in the value chain where payment is not a function of demand is between the panel and the general population respondent. Automation and access to data on costs by source could easily lead to highly personalized rewards by profile that improve participation, with the only caveat that companies will need to pay greater attention to fraud.

The third, and perhaps the toughest, challenge is providing the intangible engagement that is generally lacking in most panels. While panels have become more “social”, they don’t compete with other networks for stickiness. Whether it’s sharing how the data were used or permitting comparisons with a peer group, or offering interesting content, panel companies need to appeal to the latent feeling of “interestedness” respondents have that leads them to participate in the first place.

In his recent Greenbook article, Jan Hofmeyr reminds marketers of the importance of working on both loyalty and acquisition. This applies in equal measure to panel companies. They need to acquire and reacquire/retain participants in ways that will force them to pay attention to—or to simply pay more for/to—the people they need.

3: Suppliers need to stop putting up with abusive buyers

More has been written and said on the move to device agnostic questionnaires in the past year than any other topic related to sample.  Panel suppliers are shouting the loudest, exhorting researchers to move at a less-than-glacial pace to overhaul their designs.

Panel providers will, of necessity, start protecting their respondents. They will be able to measure survey pain by and construct “buyer ratings”, which, thanks to automation, will be easily calculable as a function of length of interview, device compatibility screenouts, abandon rate, and perhaps others. These pain measures will affect pricing as one would expect: buyers whose questionnaires fall short of the panel company’s standards will pay higher CPIs. At some level, suppliers will turn away revenue, either implicitly through pricing or explicitly by refusing to bid. This is happening already, and frankly it’s long overdue.

Similarly, companies will improve the routing experience, the process of shunting respondents hither and yon from study to study in search of a match, which still largely stinks. In part, it’s because the technology doesn’t move fast enough. In part, it’s because the industry has as yet been incapable of creating standards for passing basic demographic information, thereby requiring respondents to indicate their sex and age over, and over, and over. Many in the industry recognize the importance of standards. Work began at Samplecon and will continue at the Online Sampling Forum prior to the CASRO Technology Conference in late May.

4: Respondents will be deeply and respectfully profiled

Companies that broker survey respondents continue to explore ways in which they can add value. Profiling is one nut that many hope to crack. From micro-surveys to data enrichment to analytics that enable suppliers to infer behavioral and attitudinal propensities from survey responses, there are many exciting prospects.

Having said this, it will surely become more and more difficult to collect profile information directly. The marketing and research industries will be forced to take a far more serious approach to privacy. Privacy is a throwaway thing for researchers. The fact that panel companies generally collect this information without meaningful compensation is proof. Researchers will need to carefully watch the trends in personal data security. Mounting concerns over privacy suggest a future in which, at minimum, there’s little trust for unknown data collectors.

More intriguing are the embryonic marketplaces for personal data, one of which explicitly speaks about cutting out the market research middlemen. Entrepreneurs are helping individuals become aware of the value of their personal information and facilitate its exchange. It’s too early to know whether this will take off, but it needs to be watched.

5: Quality practices will need to evolve

Any research company of appreciable size has automated speeding and straightlining detection methods. These algorithms arose as reactions to clients’ punishing research designs. In a grid-less, device-agnostic world with shorter questionnaires, speeding and straightlining are no longer meaningful problems.

As the recent UK elections have proven, extant practices for ensuring representativity and response quality are insufficient. That the pollsters have suffered this massive a setback should strike fear into the hearts of other researchers, as the pollsters are generally more stringent in their methodology and write simpler, shorter questionnaires.

Panel companies can demonstrate additional value and potentially charge a premium by developing bias correction methods. Two avenues are worth exploring.

One is examining the relative consistency of responses. The technology exists to be able to automatically detect whether Panelist ABC’s responses are consistent with others of her demographic, attitudinal, or behavioral cohort and then decide what to do. Parameters could be set based on tolerance for some numerical deviation, at which point the respondent and his data would be eliminated or quarantined.

Another avenue of interest is asking questions that have gold-standard references. Standard practice is to weight data to correct for demographic imbalances, but more is possible. The essence of this type of bias correction would be to benchmark against gold standard data and calibrate responses to reduce bias.

Cost and Conclusions

I’ve written elsewhere that technology will, in the short term, drive down the supplier’s costs by eliminating manual labor. It will expose market liquidity and reduce the information asymmetry between buyers and suppliers, creating greater pricing parity.

But make no mistake, the marketplace that panel companies and the industry have taken for granted is that of general population research participants. The implication of the above points is that the cost of securing an in-demand respondent is likely to rise. For people with no relationship with the participant, the cost will be direct and steep. For panel companies, the cost will be the investment needed to build compelling value. They will be forced to compete like brands for loyalty and interest, which for many will be an existential challenge. This is one reason why brand-based social communities are flourishing: they are able to capitalize on pre-existing affinity.

It’s not outside the realm of possibility that the market for sample will evolve like the market for bonds. Bond quality is inversely proportional to risk: the higher the quality, the lower the risk. The only hitch in the metaphor is that higher quality bonds have lower payouts down the road, while higher quality sample will cost more up front. As mentioned above, there is (or will be) enough information being automatically transmitted to permit this sort of calculation.

It’s impossible to predict how the factors that are driving sample challenges will resolve themselves. What’s sure is that the forces are inexorable. The worst assumption the industry can make is assuming the future looks like the past. Research companies of all stripes, private and public, need to understand that they are not facing marginal cost decisions. Their actions require new thinking upon which sustainable futures can be built.

Copyright © 2015. The Author. All rights reserved.

Please share...

5 responses to “The Future of Panels:Five Investments Needed to Improve Participation and Reduce Bias

  1. One of the central points of research back in the day was validation — the notion that we could confirm what a person said and that the person was who he said he (she) was or at least locatable at the given phone number or address. Validation has virtually disappeared.

    A second point from back then was Dr. Gallup’s concern that paying respondents induces fraud.

    I’ve had clients tell me that they have tossed over 50% of online survey responses as fraudulent. I just conducted a survey on skincare in which I believe that half of the respondents were not actually qualified.

    As a panel member, I’ve seen daisy-chained invitations. “OK, you did that survey for us, here’s another one.” Frankly, that’s not sampling, much less random sampling. It’s a joke.

    We need to codify procedural standards for panel management including validation of panel members. Companies that fail to execute against these standards should be put out of business.

    Validation will drive up the price for B2B and some B2C research. However, any price is too high if you’re buying trash.

  2. JD, I loved your article and couldn’t agree more on your 5 points. I would be happy to share with you what we’ve done to build “The Panel of the Future”, offering meaningful engagement to our members. It’s been 5 years of R&D, and one of the initiatives include the usage of intrisic motivations through gamification techniques, and the development of a social-mobile application managed as a true branded community. BTW, we will be presenting at the #IIEX Competition next month and you can see a preview here:

  3. Victor, validation hasn’t disappeared. I think the user experience is broken. People are unlikely to be trustful, and times have changed from when it used to be an honor to participate in market research. As for setting standards, the associations don’t have enough power. Punishment will be delivered not by Thomas Hobbes’ Leviathan, but by Adam Smith’s invisible hand. It’s happening already. The only downside is that the waters are now really polluted with junk and it’ll take a while before they get cleaned up. There is still plenty of good, if expensive, work being done that reflects the time-honored truths of sampling. But people will continue to make choices and tradeoffs, and they will still find suppliers who are willing to take their custom.

    Adriana, I’m expecting to be in Atlanta for IIEX. I’ll stop by. I’m curious to see how you’re addressing the issues.

  4. Great article JD…pretty much all of what you covered is applicable to the qualitative universe as well. Smarter/subtler profiling is crucial to reduce participant fatigue…lengthy screening must stop if we are to recognize that the people we really want to speak with (brand passionate consumers) aren’t driven by the money but engagement with a brand or product/service coupled with a respect for their time (not that we insult them by asking for free or cheap incentives). Too many times I see clients treating participants as commodities…which baffles me given the possible connection they make with the brands being researched.

    Hope I get the chance to see you and discuss more at Ilex.

    At L&E, our panel is constantly invested in for its growth and health, and includes various means of engagement to create communities. If I could offer one suggestion to researchers (suppliers and corporate alike) to create greater engagement that would help all of us so much, it would be…see participants not as one time research subjects, but life cycle partners in the product/brand experience. Don’t just ask people to participate in a research project, share with them what you learned and once the new direction/product/service is launched, let them get a peek at what the research is going to do to their favorite brand/product. We do research with our communities all the time, and the number one thing we hear is “TELL US WHAT GOING’S TO HAPPEN FROM OUR RESEARCH PARTICIPATION.” Imagine the apostles created with your brands (and the viral marketers they would likely become for you) if before the new product was launched…you shared it with the research participants, the people that helped in the process. Recruiting partners like L&E can help with that…it wouldn’t cost much more…but would have tremendous impact not only on brands and loyalty, but also research participation and the image of the industry overall (imagine participants getting a sneak preview of a new product launch days before it came out on shelves…telling all their friends on Facebook “I participated in research for this new phone, and they listened to us and made it better by building these new features. I MADE A DIFFERENCE!” I can’t imagine a greater PR coup for brands and the research industry alike, than this.

Join the conversation