PureSpectrum - Schedule A Demo
Qualtrics: Here to Help

Purist, Meet Real World

The New York Times and CBS News recently announced the use of online panels as part of their election coverage polling. This reignited the online/phone quantitative research debate.



By Ron Sellers

When it comes to developing a purely representative sample, telephone research is still far better than online research. Numerous studies have shown this from a variety of different perspectives, leaving no real question about the comparative representativeness of the two methods.

Yes, telephone response rates are horrendous. Pew Research Center recently published a study that showed phone response rates are about 9% today, down from 36% in 1997. The Centers for Disease Control now estimates that 41% of Americans don’t have a landline (which is far higher than the dwindling proportion who don’t use the Internet).

I deal with this issue with clients all the time – should we use phone or should we use an online panel in order to get a representative sample of our target audience? My answer is that without question, telephone is the most accurate way of representing your target group…but that doesn’t necessarily mean telephone is the best way.

Quite bluntly, the issue often comes down to one major factor: cost. Typically, online surveys are vastly less expensive than phone surveys. This is true even if panel sample is selected by quality rather than just by who will throw the most warm bodies at a survey for the lowest price. The cost gap is even larger if the phone sample includes cell phones, which are considerably more expensive to dial and tend to provide even lower response rates than landlines. And the gap increases as the incidence of the study decreases.

Purists will cry “cost doesn’t matter – it’s representativeness that matters.” And to some extent, they’re right. But here’s the reality: we all have budgets. And those budgets are all limited.

Let’s say your research department (or your client) has an annual budget of $250,000 for primary research. Let’s further say that your typical study can either be conducted for $25,000 using an online panel, or for $50,000 by phone. You have two choices:

• Conduct five studies by phone and address five key issues for the year
• Conduct ten studies online and address ten key issues for the year

Puts a different spin on the phone-versus-online issue, doesn’t it? Using these figures (which admittedly are made up out of thin air), you can afford to do twice as much research if you do your work online as if you go the purist route and do it by phone.

Of course, not all studies will show this large of a gap between the costs of phone and online. But at the same time, many will show a much larger gap. Not long ago, I handled a project for a client where the incidence was less than 10%, and the sample universe was limited to the 18 – 29 age group. Out of a random-digit-dial sample, that would be about 2% of the national population. I don’t even want to think about the field costs for 1,000 completes by phone.

Now, I expect at least a few sarcastic comments along the lines of “Why don’t you just use a convenience sample at a conference, or whip out DIY, and then you can do ten times as much research for the same price?” I’m not advocating that we globally substitute low price for quality. In fact, I’m not advocating online research at all. There are times it’s worth paying for the highest quality possible, and there are certain approaches and practices that are worthless as research even if they’re free.

All I’m saying is this: cost must be one of the decision factors, because we live in the real world. If all my clients had unlimited budgets, I doubt I’d ever use another online panel. But they don’t.

In this sense, it’s similar to qualitative research. Conducting twenty focus groups is better than conducting eight. But my experience is that, unless you’re examining very different population groups or changing the stimuli from one set of groups to the next based on what you learn, twenty focus groups will provide relatively little additional benefit for more than double the cost. Yes, twenty is better than eight. No, it’s not better enough to warrant more than twice the expense.

The question for most research is not whether phone provides a more accurate, representative sample than online. That one has already been answered. It does. Without question or argument, it does. In fact, since online panels are not a random probability sample, phone isn’t just better, but the only way of providing a truly representative sample.

But the real question is this: is online panel research sufficient for what you need? Is the sacrifice in representativeness worth the ability to spend the savings on other things, such as additional studies, employee education, larger samples, deeper analysis, pre-testing, or conducting qualitative research before you do the quantitative? Can you do things to raise the quality and representativeness of the online research? (The answer to that is almost always “yes.”) At the end of the day, is the gap in statistical representativeness worth the gap in price?

In some cases, there’s no doubt that the answer is yes, absolutely – it’s worth it to spend the money for phone. In other cases, the answer will be no – online is sufficient for what you need. Not because you’re cheap or don’t care about quality, but because like everyone else, you need to take budget and time into consideration.

Until the NYT/CBS announcement, political polling was one of the last bastions of purist thought on methodology. That’s understandable, given that the polling is meant to tease out the likely winners in what are often very close races. If your candidate has 49% of the vote instead of 51%, that’s a pretty big deal. On the other hand, if your advertising awareness or customer satisfaction numbers are 49% instead of 51%, does that really impact your business model?

You need to understand enough about both methodologies to be able to make that judgment call individually for each potential project. Anyone who globally advocates one methodology over the other is missing the boat. It’s not enough just to say “phone is better” or “online is cheaper” – the balance between the advantages of each methodology is critical, as is the appropriate selection for each project.

Also critical is knowing how to get the best out of each methodology – blindly turning a panel project over to the lowest cost provider and running with whatever data results from it makes no more sense than opening the Duluth phone book, dialing a few numbers, and calling that telephone research. Panel isn’t perfect, but there are ways to get better quality data if you know how (hmmm…the wheels are already turning for another blog post).

Purists are wonderful, because their advocacy helps keep us on the straight and narrow, and makes sure we consider the hard questions. But sometimes “purist” must meet “reality” and make some tough decisions. That process appears to be happening at NYT/CBS, and it will be fascinating to see what happens as they experiment with panel sampling.

Please share...

9 responses to “Purist, Meet Real World

  1. Very good article Ron – a simple summary of the challenges faced in today’s market research industry where almost anybody can do anything at low cost and with little concern for quality.

    There is however another and important “elephant in the room” and it is the simple issue of what qualitative differences can we expect in responses from the various research medium. There has been some research done on this matter and, assuming online samples are well controlled, certain issues like personalized responses may actually yield more accurate findings.

    This has some logic attached to it since the response over a telephone may lack real consideration because often the intent is for the interviewee to get rid of the interviewer. There are some who would also argue response patterns in the Online environment have a tendency to mindlessness as seen in straight lining and other obvious quick response patterns. Clearly the tool needs to be considered carefully in any study that is not just collecting market research “racecourse data” e.g. coming in first in the Unaided Brand Awareness Stakes is ABC, followed by XYZ, etc.

    In my experience there has been low correlates across key KPI measures when we have moved personal interview trackers to an online collection medium. Makes you wonder if Online panels are just different from the total demographic market.

    The fact is everything we do with data collection options has an implication for market research. Unfortunately in these days of any information being good enough and the dumbing down of research buyers we have to ask should we just get out of the way of this irresistible force and sell out where necessary.

    I am reminded of a wonderful Chinese proverb that says “one should never play a violin in front of a cow”. Unfortunately that well may be today’s reality. The cow being …??

  2. Great stuff Ron. Being a purist may seem like taking the high ground, but the reality is that research needs to support business decisions and those decisions are made in a world with limitations of budget and time that more often than not conflict with purist ideals.

  3. Hi Ron,

    How can you say that “phone is the only way of providing a truly representative sample”, when you state that “estimated 41% of Americans don’t have a landline”. The simple truth is that there is no such thing as a truly representative population sample in my opinion. Whether phone, online, mobile or anything else is the best will depend on your objectives and yes to a huge degree cost. Each will have it’s pros and cons.

  4. Horst, there are advanced sampling practices with cell phones that approach full coverage of the US population when combined with landline sampling. Is it perfect? No. When we all had landlines, was random digit dial sampling perfect? No. There were always some people who did not have a phone at all (for instance, RDD never properly sampled people in prisons, nursing homes, university dorms, etc.). But it was as close to perfect as we were ever going to get, and phone sampling is still as close to “perfect” as we’re going to get. It’s certainly far more representative than online panels, social media sampling, or anything else. That’s why political pollsters and large-scale social works such as the CDC continue to rely primarily on phone sampling – by combining landlines and cell phones, it’s clearly the closest to a pure random probability sample, whereas the other sampling techniques are not remote close because they are not random probability at all.

  5. Probability Samples requires that all persons have an equal chance of being selected, which in today’s world is impossible to achieve. While RDD got us close in the past that is no longer the case. Even with the inclusion of cell phone, the use of call blockers, people screening calls, etc. makes the problem worse, let alone people who only use text to “talk” to others.
    You even admit that it’s not “perfect” but its as close as were going to get, you are basically saying that “yes, this isn’t a probability sample, but since I believe it’s as close as we can get to one it is perfectly okay to pretend that it is a probability sample, and equally okay to 100% dismiss all other sampling methods a not being a probability sample.”
    I think we will all be better off when we can all admit that there is no such thing as true probability sample and that while some methods may be closer than others, they are all shades of grey. We need innovation; find ways to use ALL available communication platforms to get better than the “as close to perfect” than RDD provides. No matter how “close to perfect” you think it is, you have to admit that as time goes by it is only going to get further and further away from perfect.

Join the conversation