Unearth the power of declared data
Insights That Work

Survey Respondents: The Polar Ice Caps of Market Research

I have some ideas about panels and the research ecosystem that may be hard for you to hear. Sit down, grab an ice-cold beverage and buckle up. I’m going to serve you the hard truth.

A waterfall cascades to the sea.


By Lisa Wilding-Brown

In partnership with my esteemed colleagues, I had the fortunate opportunity of developing one of the industry’s first online panels, The Harris Poll Online. Since then, I have built hundreds of panels. Mothers, gamers, business professionals, mortgage brokers, teens, high income earners, GLBTers, physicians and the list goes on. I’ve recruited every audience you can imagine. With all this in mind, I have some ideas about panels and the research ecosystem that may be hard for you to hear. Sit down, grab an ice-cold beverage and buckle up. I’m going to serve you the hard truth. I’m not doing this to win or lose friends. My goal is laser-focused. I want to see this industry survive. I want to see research budgets increase and marketers around the world to rely on our space for trusted, reliable insights for the long-term.

Let’s kick-off the conversation with a story that I think will crystalize the discussion perfectly.

Several years ago, a company that will remain unnamed, proclaimed a bold and controversial marketing message, “Panel is dead.”  At the time, my COO and other senior members of the team were outraged. “How could ‘company X’ say such a thing?!” As a result, the bold and brazen voice was silenced and our leadership relegated this firm to the penalty box.

In reality, panel wasn’t dead in the real sense. After all, the number of panels in the industry were increasing at an exponential rate. When I started in the early 2000s, there were only a couple of sample companies. In 2016, there are over 63 sample companies listed in Greenbook and I believe this number only scratches the surface. Indeed, panel is not dead, but… panel is very sick. Deathbed sick. Now hold on… let me explain before you demote me to the penalty box.

It is not unusual to hear industry bloggers and conference speakers discuss “declining response rates” or “respondent burden,” but we have to ask ourselves the question, “Do we really care?”  The short answer? ”No.” Most of the industry thinks of respondents as a commodity – an infinite resource that will always be available for our use. Plainly put: respondents are the polar ice caps of Market Research. We know they are important, but we aren’t doing much to reverse the damage of our daily abuse. Of course, we talk a LOT about the impact of long surveys, shrinking incentive budgets, hostage-taking routers, and price compression. However, at the end of the day, nothing is changing.

As a sampling professional, I feel constant pressure to reduce my pricing to compete with what I call “faux sampling companies.” These are companies that pretend to be sampling experts. Unfortunately, under the covers they are nothing more than online traffic pushers. Rain collectors with no real experience or heritage in Market Research. The market has been flooded with these types of companies over the last five years. As a sample buyer, clients must feel overwhelmed and confused by their options. These faux samplers can herd large flocks of online traffic and they know just enough MR vernacular to be dangerous. While their ability to access online traffic is impressive, we all know that sampling is much more than “driving traffic.”  Let me be clear, sampling experts can develop and replicate complex sample frames which yield representative results; traffic pushers do not.

I have to look at the bigger picture and focus on longevity and quality. “Don’t get caught up in the short-term competition,” I frequently tell myself. I have to admit, I wasn’t so strong in the past. My former CFO and board would come to me and demand savings. I would do my best to explain the long-term impacts of short-term thinking, but this fell on deaf ears. Cutting incentives by 30% doesn’t make sense. Adding routing cycles in order to increase conversion kills long-term retention. This failed routing approach produces bias and burns out respondents; and yet we see both real (and faux) sampling companies doing this every day. Enough is enough. The ice caps are melting, the water is rising and most sampling companies are drowning… or merging. Same thing.

So, how do we fix this?  How do we increase response rates?  How do we increase the pool of people who will even tolerate our surveys?  In order to improve our industry’s outlook, we need to make some significant changes across the entire research ecosystem. Here are some ideas to get you thinking…

  1. Survey design is a skill. It requires training and discipline. Just because you can program a sixty-minute survey, doesn’t mean you should. Like all things in life, we must negotiate. We must serve our clients as consultants; providing advice and evidence as to why a sixty-minute survey is an irresponsible idea.
  1. Excluding mobile respondents from your sample is another damaging idea. Stop doing it. If your survey isn’t mobile-friendly, you are doing it all wrong. Wake up from your desktop coma. Mobile is happening now.
  1. We must remember that “sample” are actually people. Let me ask you this, would you take a sixty-minute survey for $1 or the equivalent in some virtual currency? Probably not. Would you enter an endless router that holds you captive for twenty minutes only to disqualify? Probably not. We have to ask ourselves these common sense questions every day. Otherwise, we will continue to operate in our alternate universe, huddled in the fetal position waiting for our life raft!
  1. Most panels are truly broken. They are the shards of a panel manager’s broken dreams! Okay, I know what you are thinking…maybe that is touch dramatic, but it can be a depressing job at times.

Some real-life scenarios from my past life:

  1. It doesn’t make sense to change your incentive model so that it is humanly impossible for a member to redeem a reward. Yes, you will save on your monthly redemption costs, but no one will stay in your panel. As a result, your recruitment costs and outside sample spend will go through the roof.  You will spend more and save less.
  1. It doesn’t make sense to build a panel model where members only have a positive experience 10-20% of the time! Let’s distill that down for a minute: most panels only enjoy a 10-20% conversion rate (successfully completing a survey). This means that panelists are having a disappointing experience 80-90% of the time!

Who in their right mind would stick with something that produces such a negative outcome? I can tell you who: a VERY tolerant and eccentric individual who doesn’t mirror the general population whatsoever. As a sampling expert, I can tell you this is very problematic.

While another shard of an ancient glacier breaks off into the sea; you’ve likely finished your ice-cold beverage by now. I’m sorry if this blog drained you of your life force. The truth hurts, but being in a potentially obsolete industry hurts more. We have to make changes to reverse the damage and my team is setting out to do just that. We are redefining the user engagement model.  Join me in the #MRXRevolution!

Please share...

6 responses to “Survey Respondents: The Polar Ice Caps of Market Research

  1. Brilliantly written! All the new MR tech in the world can’t save us if we don’t get the basics right in the first place. Adding go the faux sample groups are those who think they can write a survey well enough on their own, then use the faux sample companies to field that survey. What would you recommend to help this particular aspect of the industry (because it’s here to stay)?

  2. Lisa, thank you for your honesty and frankness. I used to work for the original Lou Harris and Associates. I then worked on many projects with Gordon Back and the rest of his senior team while running several different research businesses. It is very sad to see what has happened to our industry over the years:. Objectivity, good response rates, accuracy etc. has gone out the window. I recently conducted a financial survey on phone, mail, and online. The weakest data accumulated in terms of accuracy and reliability was from a nationally known online panel. Thank you again for telling the truth

  3. Liza after reading your article I actually had brain freeze. An excellent and honest report of the state of things in Panel-Land. I will leave the criticism of the DIY questionnaire disruptive force to others to comment on, however panels – and here I include – community panels need a good savaging. All the talk of nurturing relationships with panelists smacks of a little desperation. Sounds good in principle but what is reality? The reality is seen in the poor response rates and the response quality e.g. straight lining, rapid completions, etc. All the talk about length of survey in the offline market as a problem pales into comparison with the dross in this industry in some markets. Thank you for an honest insiders view.

  4. Excellent article. Research used to be about people, Now, sometimes it feels as if all the technology involved means we’ve lost sight of the individuals we are trying to represent.

  5. Well, this is fascinating stuff indeed. From the qualitative side, we are seeing a lot of demands for faster and faster research (why should recruitment take two weeks or even three?) while the specs become ever more insane. The solution that panel providers are
    dangling in front of us is to use panels to recruit for qualitative work.

    Except we interact with the people recruited, even if it is online in a discussion forum or similar — and poor quality is pretty difficult to ignore in that context.

    This is a “tragedy of the commons” situation, however — participants are deemed to be an inexhaustible resource, not just for the MR Industry, but for anyone with a web site or a voice of the customer program or access to a low cost survey tool. The MR industry could fix itself without having much of an impact on the rest of this behavior, which is largely coming from people outside of the MR industry.

    I have thought for some time that the solution is to actually pay people real incentives. Yes, it will cost more. It should cost more.

    Thanks again for an excellent thought-starter!

Join the conversation

Lisa Wilding-Brown

Lisa Wilding-Brown

Chief Research Officer, Innovate MR