TMRE 2010: Projectability and a Challenge To Online Panels

by Bill Weylock, Action Insights –

First session of my afternoon: relative projectibility and sampling reliability.

REI determined that the rising percentage of cell-phone-only users makes land lines an unreliable sole-source approach method. Then they run into the problems with cell phone: more expensive to source and reach, require an incentive, not location based, ratio of cell phones to landlines not known, not able to cross off a household after reaching a number (as you can with a landline).

So on to online sampling: guess what, Anne-Marie cites the GRIT 2010 study indicating a large percentage of research professionals think online sample is less reliable than the market generally appreciates.

By the way, we are about to launch the second 2010 GRIT which will focus on new technology adoption in addition to continued tracking of basic industry metrics (with comparisons going back to 2004). The findings will be a key feature of the upcoming IIR conference on technical and technological innovation to be held in April.

Anne-Marie cited a fascinating study by Washington Mutual (just before they vanished into JP Morgan Chase) finding that respondents who take 5 or more surveys annually are less likely to desire financial products than others. Survey experience alone could account for up to 7% variation in their results. This suggests applying scoring algorithms to sample – including, among other things, extent of participation in online surveys. Panel providers take note.

This year REI did a test of methodologies, collecting data in 7 markets online and 2 markets by cell-phone while they rolled out their normal annual landline fielding. Note that cell phone interviews were only n=35 in each market, yielding directional findings only.

They found striking differences between landline and online participants in brand awareness, in ad recall and percentage of respondents who shop REI. In all cases the figures were significantly higher within the online sample.

Demographic comparisons revealed higher penetration of Caucasians and college graduates in the online sample, leading to their conclusion that online panel sample seems provide a different population than a landline sample.
One of their theories is that maybe a panel rewarded by retail incentives is more aware of and engaged with retail. In any event, they made a determination that online sample is not projectable for REI research.

Again, results are directional, but comparing landline and cell samples showed little difference in awareness, ad recall and penetration of REI shoppers. Demographic analysis suggested that cell phone sampling could also provide reach into populations hard not adequately delivered by landlines.

Next year REI is going to test ABS with landline and cell phone data collection. They can match sample to cell or landline numbers at about 65% – 75% and expect that this will become their sampling method of choice going forward.

It seems that REI has rejected online sample and is harking back to a previous era in bringing ABS to the fore for 2011. It is an evolving experiment, but they expect that ABS with sampling shared by cell and landline phone outreach should provide more representative sample than previously obtainable.

Okay, online sample providers. I did ask.

First I asked Anne-Marie whether she was saying that online sample is unreliable and unprojectable and should be rejected by all market researchers in favor of phone. It didn’t come out exactly that way, but that was the gist. I wondered (and frankly was hoping) to hear that there was something peculiar to REI that made the demographics of online sample unrepresentative of their universe.

Unwilling to go as far as a universal slap at online sampling, Anne-Marie simply restated her opening thesis: that marketers should ensure that their sample is sufficiently representative of their markets. In REI’s case, online sample does not seem to hit the mark.

I also asked, why she had been unable to weight the online sample to census. The rub seems to be that the panels were unable to provide sufficient penetration in key markets to support weighting.

This topic seems to cry out for more coverage and certainly more discussion. To my ear it is as provocative as anything we have heard in the past two days. What do you think?