Research Methodologies

August 25, 2014

Purist, Meet Real World

When it comes to developing a purely representative sample, telephone research is still far better than online research.

Ron Sellers

by Ron Sellers

0

reality-check

 

By Ron Sellers

When it comes to developing a purely representative sample, telephone research is still far better than online research. Numerous studies have shown this from a variety of different perspectives, leaving no real question about the comparative representativeness of the two methods.

Yes, telephone response rates are horrendous. Pew Research Center recently published a study that showed phone response rates are about 9% today, down from 36% in 1997. The Centers for Disease Control now estimates that 41% of Americans don’t have a landline (which is far higher than the dwindling proportion who don’t use the Internet).

I deal with this issue with clients all the time – should we use phone or should we use an online panel in order to get a representative sample of our target audience? My answer is that without question, telephone is the most accurate way of representing your target group…but that doesn’t necessarily mean telephone is the best way.

Quite bluntly, the issue often comes down to one major factor: cost. Typically, online surveys are vastly less expensive than phone surveys. This is true even if panel sample is selected by quality rather than just by who will throw the most warm bodies at a survey for the lowest price. The cost gap is even larger if the phone sample includes cell phones, which are considerably more expensive to dial and tend to provide even lower response rates than landlines. And the gap increases as the incidence of the study decreases.

Purists will cry “cost doesn’t matter – it’s representativeness that matters.” And to some extent, they’re right. But here’s the reality: we all have budgets. And those budgets are all limited.

Let’s say your research department (or your client) has an annual budget of $250,000 for primary research. Let’s further say that your typical study can either be conducted for $25,000 using an online panel, or for $50,000 by phone. You have two choices:

• Conduct five studies by phone and address five key issues for the year
• Conduct ten studies online and address ten key issues for the year

Puts a different spin on the phone-versus-online issue, doesn’t it? Using these figures (which admittedly are made up out of thin air), you can afford to do twice as much research if you do your work online as if you go the purist route and do it by phone.

Of course, not all studies will show this large of a gap between the costs of phone and online. But at the same time, many will show a much larger gap. Not long ago, I handled a project for a client where the incidence was less than 10%, and the sample universe was limited to the 18 – 29 age group. Out of a random-digit-dial sample, that would be about 2% of the national population. I don’t even want to think about the field costs for 1,000 completes by phone.

Now, I expect at least a few sarcastic comments along the lines of “Why don’t you just use a convenience sample at a conference, or whip out DIY, and then you can do ten times as much research for the same price?” I’m not advocating that we globally substitute low price for quality. In fact, I’m not advocating online research at all. There are times it’s worth paying for the highest quality possible, and there are certain approaches and practices that are worthless as research even if they’re free.

All I’m saying is this: cost must be one of the decision factors, because we live in the real world. If all my clients had unlimited budgets, I doubt I’d ever use another online panel. But they don’t.

In this sense, it’s similar to qualitative research. Conducting twenty focus groups is better than conducting eight. But my experience is that, unless you’re examining very different population groups or changing the stimuli from one set of groups to the next based on what you learn, twenty focus groups will provide relatively little additional benefit for more than double the cost. Yes, twenty is better than eight. No, it’s not better enough to warrant more than twice the expense.

The question for most research is not whether phone provides a more accurate, representative sample than online. That one has already been answered. It does. Without question or argument, it does. In fact, since online panels are not a random probability sample, phone isn’t just better, but the only way of providing a truly representative sample.

But the real question is this: is online panel research sufficient for what you need? Is the sacrifice in representativeness worth the ability to spend the savings on other things, such as additional studies, employee education, larger samples, deeper analysis, pre-testing, or conducting qualitative research before you do the quantitative? Can you do things to raise the quality and representativeness of the online research? (The answer to that is almost always “yes.”) At the end of the day, is the gap in statistical representativeness worth the gap in price?

In some cases, there’s no doubt that the answer is yes, absolutely – it’s worth it to spend the money for phone. In other cases, the answer will be no – online is sufficient for what you need. Not because you’re cheap or don’t care about quality, but because like everyone else, you need to take budget and time into consideration.

Until the NYT/CBS announcement, political polling was one of the last bastions of purist thought on methodology. That’s understandable, given that the polling is meant to tease out the likely winners in what are often very close races. If your candidate has 49% of the vote instead of 51%, that’s a pretty big deal. On the other hand, if your advertising awareness or customer satisfaction numbers are 49% instead of 51%, does that really impact your business model?

You need to understand enough about both methodologies to be able to make that judgment call individually for each potential project. Anyone who globally advocates one methodology over the other is missing the boat. It’s not enough just to say “phone is better” or “online is cheaper” – the balance between the advantages of each methodology is critical, as is the appropriate selection for each project.

Also critical is knowing how to get the best out of each methodology – blindly turning a panel project over to the lowest cost provider and running with whatever data results from it makes no more sense than opening the Duluth phone book, dialing a few numbers, and calling that telephone research. Panel isn’t perfect, but there are ways to get better quality data if you know how (hmmm…the wheels are already turning for another blog post).

Purists are wonderful, because their advocacy helps keep us on the straight and narrow, and makes sure we consider the hard questions. But sometimes “purist” must meet “reality” and make some tough decisions. That process appears to be happening at NYT/CBS, and it will be fascinating to see what happens as they experiment with panel sampling.

0

online researchphone surveysrepresentative samplerespondent engagement

Disclaimer

The views, opinions, data, and methodologies expressed above are those of the contributor(s) and do not necessarily reflect or represent the official policies, positions, or beliefs of Greenbook.

Comments

Comments are moderated to ensure respect towards the author and to prevent spam or self-promotion. Your comment may be edited, rejected, or approved based on these criteria. By commenting, you accept these terms and take responsibility for your contributions.

More from Ron Sellers

Are the Fraudsters More Sophisticated Than the Researchers?

Research Methodologies

Are the Fraudsters More Sophisticated Than the Researchers?

It’s amazing what some people will do in order to make a buck-fifty. Two recent studies have brought to light how sophisticated panel fraud has become...

Ron Sellers

Ron Sellers

Still More Dirty Little Secrets of Online Panels

Research Methodologies

Still More Dirty Little Secrets of Online Panels

Nearly half of your panel data is trash. Here is how to fix it.

Ron Sellers

Ron Sellers

Can Political Polls Really Be Trusted?

Can Political Polls Really Be Trusted?

When political polls fail to predict the exact outcome of an election, maybe they’re not wrong…maybe we are.

Ron Sellers

Ron Sellers

Panel Quality Stinks and Clients Are To Blame

Research Methodologies

Panel Quality Stinks and Clients Are To Blame

Why should panel companies improve their results when clients accept the status quo and won’t pay for better?

Ron Sellers

Ron Sellers

ARTICLES

Moving Away from a Narcissistic Market Research Model

Research Methodologies

Moving Away from a Narcissistic Market Research Model

Why are we still measuring brand loyalty? It isn’t something that naturally comes up with consumers, who rarely think about brand first, if at all. Ma...

Devora Rogers

Devora Rogers

Chief Strategy Officer at Alter Agents

The Stepping Stones of Innovation: Navigating Failure and Empathy with Carol Fitzgerald
Natalie Pusch

Natalie Pusch

Senior Content Producer at Greenbook

Sign Up for
Updates

Get what matters, straight to your inbox.
Curated by top Insight Market experts.

67k+ subscribers

Weekly Newsletter

Greenbook Podcast

Webinars

Event Updates

I agree to receive emails with insights-related content from Greenbook. I understand that I can manage my email preferences or unsubscribe at any time and that Greenbook protects my privacy under the General Data Protection Regulation.*