Research Now SSI: Scale Matters, Accuracy Matters, Confidence Matters
Track Opinion
Shopper Intelligence

An Open Discussion on the Impact of Respondent Sourcing

Frank Kelly explains how the quality of your research relies on the source of your respondents.

By Frank Kelly

slide-5

Research clients should learn how the source of their respondents affects research data. At Lightspeed GMI, we have been observing some very significant differences depending on the source of the data. The way a respondent enters a study will impact the overall data quality obtained from that respondent. It is not a story of good and bad, but rather just a case of significant differences; ignoring these differences could result in poor quality research.

Clients still believe the respondent world is a choice between research panel respondents and river respondents. In reality, the river sourcing technique has not existed for several years. What we have instead is a range of panel types that mostly have nothing to do with research; members of these panels need to do something in order to get something they want, in this case, complete a survey. The important factor here is that river used to indicate ‘freshness,’ which was seen as a virtue when compared to research panel respondents that were seen as ‘conditioned.’ A dynamically sourced respondent (which is what we call the process of amalgamating non-research panel respondents) actually takes many more surveys than a research panel respondent.

The majority of respondents sourced from most panel companies are actually dynamically sourced respondents that come from a variety of places (i.e., social media sites, game sites and loyalty sites).  These are often mixed with respondents sourced from traditional research panels. The key point here is that we get very different data depending on the source. Dynamic source respondents consistently rate product concepts higher, they consistently have a higher incidence of qualifying for studies (due to over-claiming behavior) and they are less attentive survey takers. We can fix attentiveness and over-claiming through our data quality process, but the difference in the way they answer questions cannot be fixed.

At Lightspeed GMI, we believe that research panel respondents are the preferred method for your research needs, but we also feel that a small amount of dynamically sourced respondents can be beneficial for most studies. We normally recommend an 85% panel/15% dynamic source blend. This blend enables us to take advantage of clear dynamic sourcing benefits: it is inexpensive and has complimentary demographics to panels (lower income, larger households, less education and more ethnic populations). If we blend at this level, we tend to stay closer to category benchmarks than at higher levels of dynamic sourcing.

Consistency is key. You should determine if you will be using research panel respondents, dynamically sourced respondents or a blend. The ratio of these sources should be help consistent for all quota cells in the study and if the study is a tracker or wave study, the blend should be consistent form period to period.

Please share...

15 responses to “An Open Discussion on the Impact of Respondent Sourcing

  1. I honestly did not know how to respond to this at first – a panel company suggesting using sub-samples of dynamic respondents in a mix with panel respondents. This discussion actually raises the bigger issue which I have been making for years that panel respondents are indeed different from recruited face to face respondent samples and now we find out from “dynamic” sourced respondents. One observation noted is that these dynamic respondents seem much more involved with media communications. Well in my experience that is the exact same observation noted between face to face recruited samples and your basic panel respondent. In a number of cases where I was involved with moving continuous tracking studies from the offline to the online we consistently found higher awareness measures across brands and ads. This struck me as a simple fact – panel recruited sources are just different to the mass market. I believe a recent RiWi studied using unincentivized respondents also highlighted significant response differences with matching panel samples. It’s a bit strange therefore to think that a sub-sample of dynamic panelists will provide some corrective effect. In my view they will only further overstate what are more accurate performance measures from non-panel data. Of course the research buyer does not want to know about this because the cost differential in favour of online data collection is more than enough to offset any slippage in objectivity. I don’t even want to get into any discussion on the rhyme or reason for a 15% incidence. Where did that come from? Even the most reasoned evaluation would suggest that any dynamic sample selection inclusion is just adding more bias to what is already biased data across a number of key brand performance measures.

  2. Very thoughtful and interesting post! It is true that the choice of the sampling method (F2F, CATI, Online Panel, Mobile App, Dynamic, Mail etc.) has an impact on response behaviour.

    We see every day, though, that a far bigger impact on response behaviour and data quality actually comes from the survey itself. A poorly designed survey yields poor results; a well-designed survey yields excellent results – no matter what channel is used to reach people.

    Writing better surveys will get us a long way towards improving quality and attracting people to participate in research who would normally not do so (which is probably more than 95% of a population).

    With regards to methods, I think another exiting question is the following: there will soon be over four billion people running around with a smartphone. How can we use this unprecedented reach to create faster and better insights around the world (and thereby transform our industry)?

  3. I’m new to the sampling industry terminology: what is river sample, and what is dynamic sample? Is dynamic in this sense another panel strategy? And echoing Chris’ question — what is the data to back the claim of the 85/15 mix? Another follow-on question: if working on a large tracker study that has traditionally used 100% panel sourcing, and you want to modify for current or future waves without sacrificing data quality (especially a sudden increase in sat scores as a simple result of changing sample composition), how would you do that? Or would you simple stick with the sample comp as it’s been done previously?

    Nico – spot on re: tapping into mobile. I think we need to stop thinking of adapting existing online surveys to make them fit mobile, and start creating surveys with mobile in mind first. That means shorter, different q-types, no more matrix questions!

  4. Perhaps what is more important here is to recognize that no matter WHERE you source research participants, a nonprobability sample will never be a probability sample. If you focus on one panel provider, you are biased to their sources. If you focus on a dynamic source, you are biased to those sources. Assuming all take great care with their quality processes, none is “better” than the other. Each one represents a unique slice of the population that cannot be replicated except by continually sampling only from it. So the issue is this. Use the right sample for the appropriate purpose, and focus unwaveringly on data quality from a total quality perspective.

  5. Thank you, Frank. You’ve said what we’ve been saying for years. Sourcing matters. Consistency is key. Quality is worth it. Including as many different types of people in research as possible ensures better data. It’s really very important that companies work with large, established partners with a very broad reach of survey takers. When we do this well, the data represents the populations. When we don’t, it doesn’t.

    I also agree with two comments about quality throughout the process. Great sampling is lost when the survey instrument is bad. As an industry this is our kryptonite. Every time we get near this topic of survey designs, the conversations dies. We absolutely must figure out how to ask questions people can answer and protect them from things that chase them away.

  6. Thanks Frank for summarizing the most impactful presentation I had the chance to view at the IIex. I applaud that you pointed out that river (now referred to as dynamic), as we understood it (a source of fresh respondents), has not existed for years. This is something many client-side users are not well informed of and something very few panel suppliers are willing to talk about. I was really floored by one slide you presented in particular that gave me a real insight of where the panel industry is these days. The questions themselves surprised me and the answers should give anyone who monitors data quality pause.

    The slide I am referencing:

    – The average minutes prior to this survey spent answering survey questions today (Panel 16.4 mins. – Dynamic 28.8 mins) – this is incredible, and this is only a point in time during the day measure, not a total time spent during the day.
    – The average number of online surveys completed per month for any research organization (Panel 30.4. – Dynamic 41.2) – so if an average survey is 20 minutes (my estimate) that means panelist do, on average, 10 hours a week of surveys and a dynamic survey taker spends almost 14 hours a week. How do you focus on quality after this? This is a broken system.
    – The average number of panels received communication from in past 6 months (out of 27 panels) (Panel 6.5. – Dynamic 8.0)

    Kudos to the routing technologies and programmatic buying for creating a highly efficient way to create dynamic sample, but I fear the result has been an over sharing of a very limited supply of people. To Melanie’s comment that “It’s really very important that companies work with large, established partners with a very broad reach of survey takers.” I would suggest that your slide and other studies that have been made public, demonstrate that large companies do not have a broad reach but rather a small fraction (much less than 5% of a population would be my guess) of professional survey takers, which are broadly shared with other large companies. Consistency is not the key if you are consistently over measuring a small, unique group. Perhaps large companies that are built on the current model aren’t those who we should be looking to for answers.

  7. Chris.

    To clarify your point “I believe a recent RiWi studied using unincentivized respondents also highlighted significant response differences with matching panel samples”. We did not compare our data to panel data, but rather to what we consider “frequent” responders (have done a survey in the last month) to those who are not frequent survey takers (have not done a survey in the past month or have never participated in a online survey). We get 66% – 85% of our data from people who have not done a survey in the last month, depending on the country. We saw significant differences in 19 of 21 questions we asked. I agree with your long-held belief that people who participate in surveys 14 hours a week are different in many aspects to the vast majority of the online population should be common sense.

  8. Grant thanks for correcting. I guess the conclusions still kind of make the point that panel participants are in some ways different from regular unincentiviized (most representative?) samples. The data from the ILEX presentation by Frank Kelly even heightens my concern. The richness of the responses and participation levels suggests even more strongly that any form of panel recruitment will result in a “biased” sample. As tracking clients we can live with that because at least we know continuity of sample is a given. The transition from offline to online is problematic. We typically ran a 4-6 weeks data overlap and merged the data. Some measures needed a forced correction of history to make tracking data look sensible over time.

  9. While I’m correcting I should correct my math. It’s 10 and 14 hours per MONTH not per week. It’s it still cumulatively a full day or two a month of full time survey taking, with a seemingly daily habitual-like schedule.

  10. Good discussion that I appreciate all providing commentary have provided…being from the qualitative sphere, its interesting from my point of view how sampling continues to be the primary subject, regardless of qual or quant. It also begs the question, given panel companies are now by and large doing the recruitment for online qualitative technology companies, how this affects qualitative research as well? Is the participant different, given the panel is the same dynamic source, and could their responses be equally flawed given the biases of their type?

    Qualitative companies (facilities) certainly have their own demons to exorcise, but for those that are investing in their panel development and technologies to manage them, perhaps its time to rethink who’s participating in our research, and where they come from, given the only companies in the US that employ local in person tactics to build panel are in fact companies with local presences (i.e. facilities).

    Who is the participant, how the participant was found, how the research was administered…I just recently watched Greenbook’s preso of the CEO of Kantar’s speech at Ilex in Europe. He spoke of research transforming from farming to chef…it seems most respondents to these discussions feel corporate researchers don’t value the “ingredients” and expertise of the farmer delivering those ingredients (or the chef, ergo the questionnaire/screener, cooking with those ingredients), they just care about speed and cost. I don’t see how we can increase our value to the c suite if the quality of the meal we deliver values satisfying the immediate hunger over a truly memorable and impacting meal/experience.

  11. dear friends, I have followed with great interest the discussion started by Frank Kelly and it seems to me that you have raised some important issues that deserve some comments and inspire actions. Comments first.
    (1) No access panel can claim to represent the entire web population. True. But the same is true for other sampling sources. Remember that in telephone surveys you get 80% rate of refusal or not at home. In face to face intervies (when you prescrbe a formal random selection of respondents) the rate of refusal is even higher. The old “mal surveys” (do you remember them?) had a rate of responsen in the range of 5-10%.
    (2) This problem has Always been there, whatever samplig procedure were followed and quantitative researchers had Always had to cope with it )even qualitative, listening to Brett Watkins, but they don’t bother with standard error nor confidence intervals). The problem has Always been there but now we are in a better position to solve it.
    (3) Better? Really? Yes! Everybody among us has access to some clear evidence. If we agree to share, sincerely, our piece of evidence we could do important progresses and present better samples (or keeping the sampling error under control). Some examples:
    (4) four years ago we did a political survey in Milano (1.7 million inhabitants). otal sample 1.200 citizens, sampled through 4 access panel providers: Toluna, SSI, VIA! (our own panel) and GMI (at that time not yet Lightspeed). The differences were amazing and also instructive.
    We reported this experienca at a MRA workshop in California, but with many limitations, because you cannot go public with all these results.

    Hence my proposal.

    When we ask to opt in in our panel our rate of success is less than 20%. And from these our first survey will have 35% of rate of response. So our coverage is around 7%. What do we know about the remaining 93%? A lot, if we take this problem seriously. We did a lot of experience in this area and we are ready to share our results to other professionals interested in this subject. Of course in a confidential site or among a closed Group. Is anybody interested?
    Thank you anyway for inspiring discussion
    Carlo Erminero

  12. Carlo, what a great idea. If only there was a forum for sharing insights. My premise is that at the end of the day reality does bite and we have to live with sample bias in any form. I just feel we ought to know more about those differences. We do seem to have evidence that a panel source of any kind tends to be a more marketing communications engaged target,

  13. Thanks for the river of relevance and reason!-) As a newbie to this site, but longtime ad agency owner, I’m finding very interesting stuff here. I could have really used this particular article and thread a couple of years ago to help educate one of our clients on the matter of sample sourcing. It’s also something I’ve written about based on our own experience if anyone is interested. http://www.robinashmore.com. In any case, I’ll be reading you and following along.

  14. Dear friends in panels,
    the issue of source in our sampling is of paramount importance to support validity and reliability of our results. For every survey the selection of the sample from an access Panel can be fully random, nearly perfect, according to theory. The problem is in the source, i.e. the composition of the Panel itself. No access panel in the world can claim to “represent” the web population in the Country, nor any properly defined big segment. Hence our problems. True. But I would like to invite my colleagues researchers to accept the following two statements, either:
    • the same is true (or even worse) for any other source of sampling
    • the problem has always been there, but now we are in a better position to solve it.

    Every Access Panel Provider has (or could easily have) a lot of evidence in his archives. We have. And we intend to go in deep with analysis and share results with our colleagues in this blog or elsewhere, if somebody else is interested and ready to do the same. Do you manage an online Access Panel in some countries? Or are you frequent user of online access panel for your research on general population?

    I iwould like to invite all of you to show and comment your experience about reliability and validity of your results. In my opinion this will be the way to take seriously the invitation of Frank. Everybody should show advantages and limitations of his stuff. This will allow for an happy coexistence of so many sampling techniques and sources, like we have now. Google, Research Now, GMI … have different practices and the Client should be put in condition to find out the differences. So decide what is more suitable for his problem.
    There is another possible precious outcome of these efforts: to find a specific “design factor” that could correct our standard calculations of errors, confidence intervals and the like. The Frank’s recipe 85% from permanent Panel plus 15% from river sampling is just the first step on this road. It should be supported by more technical proof. But he showed the way.
    Next steps? If somebody is interested out there, please let me know!

    Thanks to everybody

    Carlo

Join the conversation