Categories
Research Methodologies
November 8, 2017
While research companies are pushing for speed in turn around time, often respondent and data quality is being sacrificed.
Business solutions in 48 hours! Get your survey data overnight! Do agile research! Fast, faster, fastest!
Yes, it seems the insights world is moving faster and faster every day. Many companies are promising turnaround times that would have seemed absurd just a decade ago. Shorter questionnaires, automation, and DIY solutions all offer speed and more speed.
But there’s one big question with this race to be faster than everyone else: what’s getting sacrificed?
No matter how a questionnaire is designed or how data processing or reporting are automated, there’s still an important component to any quantitative study: respondents. And while online research panels can give you access to thousands of respondents in just hours, panel quality ain’t gettin’ any better, folks.
As regular users of panels, we are also regular recipients of bad respondents mixed in with the good ones:
But aren’t panel companies and field agencies screening out the bad respondents for you? Well, they’re trying, but many of their solutions are automated (again, in the interests of being cheaper and faster). For example, they’ll employ an algorithm that automatically tosses any respondent who answers a questionnaire in less than 50% of the average length, or one that catches straightliners in all your grids (that is, if you’re still using lots of grids).
Frankly, they just miss a lot.
Panel quality is atrocious today. Grey Matter Research has adopted the position that every respondent we get is a bad respondent, until we can demonstrate otherwise. This takes a lot more than digital fingerprinting or pre-programmed algorithms. Usually, it requires going line-by-line through the data to find and remove problem respondents. Just a few ways we do this:
Of course, respondents do make mistakes or misread questions, so usually the decision to toss a respondent is from a combination of factors. They straightlined the one short grid we included? Mark ‘em yellow. They further completed the 12-minute questionnaire in 8 minutes? Downgrade to orange. Also answered the question “What are the main reasons you are not at all interested in learning more about this product” with “I like this advertisement the best”? Buh-bye.
So what does any of this have to do with speed? (Or with brownies…but I’ll get to that in a moment.) Simple: this cleaning process is not a fast one. It doesn’t have to take days, but it won’t be done in minutes, either. In the quest for getting your data faster, how many of the respondents you’re getting are bots, duplicates, satisficers, or those who just didn’t actually pay attention to the questions you were asking?
Do you have any idea how many respondents had to be replaced on your last study? Or what criteria your vendor used to identify fraudulent or poor-quality respondents?
Most importantly: Did your vendor even do anything beyond some basic, automated checks to assure you got real, quality respondents?
Make no mistake – this is not just a problem with quick turn-around surveys. I’ve seen plenty of databases delivered in no particular hurry that still lacked proper quality control. But going all-out for speed dramatically increases the chances that your data includes some bad respondents, because putting everyone on a rush basis makes it far less likely that there will be time available for quality control.
In a qualitative interview last month, I had a respondent object to a product concept, because she felt one small part of the statement was not true. When I probed for why this undermined the whole concept, she earthily explained, “Even a little bit of poop in the brownie batter means I’m not going to eat the brownies.”
So what proportion of bad respondents are you willing to accept in order to get your data faster: 2%? Five percent? Ten percent? Twenty?
Or, to paraphrase my favorite respondent of the year so far: How much poop will you accept in your batter in order to get your research brownies baked faster?
Comments
Comments are moderated to ensure respect towards the author and to prevent spam or self-promotion. Your comment may be edited, rejected, or approved based on these criteria. By commenting, you accept these terms and take responsibility for your contributions.
Disclaimer
The views, opinions, data, and methodologies expressed above are those of the contributor(s) and do not necessarily reflect or represent the official policies, positions, or beliefs of Greenbook.
More from Ron Sellers
It’s amazing what some people will do in order to make a buck-fifty. Two recent studies have brought to light how sophisticated panel fraud has become...
Nearly half of your panel data is trash. Here is how to fix it.
When political polls fail to predict the exact outcome of an election, maybe they’re not wrong…maybe we are.
Why should panel companies improve their results when clients accept the status quo and won’t pay for better?
Sign Up for
Updates
Get content that matters, written by top insights industry experts, delivered right to your inbox.
67k+ subscribers