Our new GreenBook Directory site is live!
Your #1 strategic guide to consumer insights.
Qualtrics X4 The Experience Mgmt Summit
Brand & Retailer tickets for all IIeX events now start at just $99! Get or give one today!

Does (Panel) Size Matter?

We announce the size of our panels to show scale, and to impress clients. It’s how the industry has defined success over the past decade; however I want to offer up another metric for your consideration: traffic.


Editor’s Note: USamp is one of several firms that are redefining the model for what a panel provider looks like in the new MR paradigm. Their incorporation of self-service survey tools, a mobile platform for data collection, and the merging of both traditional panel, social sample, and river sampling has set the stage for not just a new panel business model but also a whole new series of questions on what metrics should be important in judging panel partners. One of those is how do we factor in traffic metrics into a portal-based sampling model. It’s certainly something that is an increasingly relevant consideration and I suspect it will only increase in importance as new research models emerge and take hold.

Matt Dusig, CEO of USamp has been challenging the industry to look at these new ideas and I think it’s a conversation that we need to have industry wide. Not only do these questions apply to panels, but also to social media research, gamification, mobile, MROCs…heck, just about ANY newer research approach is impacted by these ideas. This isn’t meant to be a uSamp advocacy post; I think the questions Matt raises are important and pertinent and I’m happy that he asked me to guest post this on GreenBook Blog. I’ll be very curious to see what type of debate this generates!


By Matt Dusig, CEO of USamp

Most sample companies market themselves according to panel size, quality of respondent data, variety of the traffic sources and customer service excellence. We announce the size of our panels to show scale, and to impress clients. It’s how the industry has defined success over the past decade; however I want to offer up another metric for your consideration: traffic.

I think the new discussion shouldn’t only be about panel size, but about traffic and transparency. As sample companies move beyond the pure process of building panels, daily/monthly web traffic becomes a core metric. Traffic is a foreign term to most in the market research industry. Traffic = website visitors. What good is a website or any online business without visitors? But what good are visitors without the right technology to manage the entire process? From registration to rewards, from deep profiling to quality measures, from routing technology to creating a positive experience—all require excellent technology. And if we don’t honestly and transparently convey the entire process to our clients, then we’re doing our clients and ourselves a disservice.

So how do we measure traffic? Why is traffic important?

Let’s say you want to recruit people from Facebook to take a survey. You don’t really care about the fact that Facebook has a database of 750 million people. You care about how many people will be exposed to your ad during the days your survey is in field. It’s all about visitors and traffic.

When you want a sample company to find panelists for an upcoming survey it doesn’t matter how big the panel is; it matters how people will respond to survey invitations and how much traffic will visit a sample company’s sites during the days you’re in field.

I think the real hard-hitting questions to ask sample companies are as follows:

1)  Traffic:

a. How much traffic do your websites see DAILY?  

b. What % of traffic is US vs. International?  c. Of the daily visitors, what % are new visitors vs. returning visitors? 

2)  Completion and Experience:

a. What % of your daily visitors who log into your site successfully complete a survey on that day?

b. What % of people who click from an email invitation successfully complete the survey they were invited to?

c. How long is the routing process for a panelist (the process of bouncing a person from question to question before they get routed into the appropriate survey). 

3)  Activity and Quality:

a. How do you define an active panelist?

b. What do they have to do to stay active? 

c. If someone isn’t joining your panel, how do you monitor activity to maintain quality? 

All this talk about traffic and transparency is not to underestimate the importance of size. But I want to recognize that there are more dimensions to a panel than size—and more important questions that clients should be asking their suppliers. Clients should be very concerned if sample vendors aren’t willing to share their traffic sources and all panel attributes. They have a right to know what they are purchasing. I invite fellow suppliers to join uSamp’s mission to make the market research industry more transparent. Who’s ready to reveal?

Please share...

4 responses to “Does (Panel) Size Matter?

  1. I don’t get this discussion about panel size X traffic.

    In the “off-line world” , sampling methodologies such as street intercept, mall intercept, door-to-door, HH panels, and “convenience sample” methods are well known and broadly used by the MR Industry.

    In the “digital world”, river sampling (the online version of street intercept), recruitment in social networks (the biggest venue where large numbers of target groups can be found), online access panels, mobile panels are just variations of new sampling methodologies becoming more common and accepted by the MR industry.

    So, at the end of the day, panel companies should be able to execute any of these new sampling methodologies (using digital channels, social networks, etc.), if they want to survive nowadays.

  2. Agree with what Adriana says. Plus obviously all of our other old-fashioned MR sampling tools like quotas etc. may or may not apply, depending on what we are looking for.

    In terms of “does size matter” – well yes and no. Most of the time we are looking for a specific target group and/or want our sample as representative to your target population as possible. So for a generic representative survey or even a tracker sample size matters BIG time. For a specialist target group, say GPs, sample size doesn’t matter so much, it’s more important to get to them.

  3. When you have a universe that selects itself and respondents that selects ithemselves, an incentive (however menial) and panelists who are members of 3 or 4 panels completing 2 surveys a week for each panel, it doesn’t really matter about size, the quality of the insight is a lottery. Fortunately most of our clients don’t buy into the ‘we can’t afford to do it properly so lets use a panel’ argument.

  4. Hi Mike and Horst,

    Thanks for your comments. I agree with Horst that size can matter or not, but, as I commented before, panel companies have developed great capacity to reach, connect and recruit huge number of people to participate in surveys. I forgot to mention in my previous post the usage of large global affiliate networks and survey routers as other common recruitment methodologies used by the global panel companies as well. So panel size and reach capacity is not an issue any more.

    In the last 5-6 years, the industry has done a great job developing quality control procedures to eliminate professional respondents from panels, de-duplicate sample, validate respondent profiles, eliminate speeders, or bad respondents from surveys, and many other technological tools and methods to assure sample quality. So I’ll have to disagree with Mike. In my opinion, the main issue concerning quality of insights nowadays, when using online methodologies, is not related to the panels or respondents themselves, but with the researchers. The truth is that they haven’t learned yet that respondents are not willing to spend 30-40 minutes answering poor surveys. I can say this after 11 years working in this industry in a daily basis and seeing thousands of online surveys being developed as poor digital versions of their off-line ones.

    Consumers have changed dramatically. They have access to information and communicate with other people instantly, interactively, and globally. They use mobile devices, social networks, blogs, instant messengers, forums, and many other interactive channels to stay connected, informed, and to share knowledge, experience and collaborate with others. That is the way they would expect to interact with researchers, sharing their opinions, knowledge, experience and the best quality insights we could ever imagine get and provide to our clients.

    The MR Industry needs to wake up before it’s too late.


Join the conversation