Let’s Stop Talking About Mobile Surveys

The latest GRIT (Q3-Q4 2017) report states it unambiguously:

“We need to talk about sample quality. Issues surrounding accessing quality and representative sample is [sic] without doubt the single biggest individual challenge mentioned in the survey.”

If we sift through the open ends in the report for those relating to sample, we find three key themes:

Feasibility and representativeness

• “Obtaining a representative sample”
• “Lack of probability samples”
• “Decline in response and participation rates across ALL modes and methodologies”

Real, honest humans

• “Frustratingly low quality”
• “More and more professional respondents”
• “Fraudulent responses, whether generated by individual respondents for personal gain, or on a larger scale by sophisticated bots”

Engagement

• “Apathy, fatigue and disinterest makes it difficult to generate reliable, sincere and in-depth answers”
• “Getting quality respondents that meet client criteria to stay engaged over the long term”

These issues will come as a surprise to no one, as they have occupied a meaningful part of the industry’s public discourse for years.

The solutions proposed in GRIT are familiar as well:

Better surveys

• “We have to improve surveys.”
• “Need to cap survey length at 15 minutes and make sure all surveys are mobile friendly.”

You get what you pay for

• “More training for DIY platforms”
• “Stop offshoring”
• “Pay panelists more”
• “Clients perceive research sample as a commodity, and almost always buy on price”

Let’s build a campfire and sing songs

• “Come together as an industry”
• “Research vendors should be more transparent with some of the potential issues”
• “Industry initiative with the involvement of all stakeholders”

The best one can say about any of these solutions is that they have been unsuccessful. Despite massive amounts of education, including everything from research-on-research to parading real panelists into conferences to talk about how they game the system, long poorly-written desktop-only surveys are still too common. Bemoaning the advance of technology or changes in the market is like shouting at the sun for rising each morning. Anything approaching binding industry standards is a pipedream for suppliers and buyers: the businesses are too diverse and the stakes are too high.

The worst aspect of these solutions is that none of them are forward looking. They arise from an antediluvian understanding (think: 2007!) of the research market. Equally troubling is that these comments don’t speak to ways in which sample suppliers share responsibility for the blame and solutions.

Forward-looking solutions

As an industry, we do ourselves a disservice by not thinking about these problems in new ways. Here are some possible solutions which are both forward-looking and reason for hope.

First, we need to do a much better job defining what we mean by a bad experience.

Time limits, though important, oversimplify the problem. For example, we have known for four years (my former NPD colleague, Inna Burdein, presented this at CASRO in 2013), that people will be happy with longer simple surveys than shorter harder ones. We also know that incentive levels, question format, response types, demographics, topical interest, day of the week, time of the day, a host of other elements impact the respondent experience–and this does not include very common issues with project setup, quota management, and programming!

From field conditions to participant data to engagement tests, there are dozens of variables we can use to evaluate the quality of the respondent’s experience. We should also be allowing the respondent to comment as well. Survey ratings can capture this. If a respondent tells us a survey is great or it stinks, we should use that data. When all this data is combined with a supplier’s economics, actuarial techniques can bring more clarity to what constitutes a bad experience, which can continue to be shared with buyers.  

Second, we need to go all-in on technology and automation for sampling

Here’s the thing: the situation is bad enough that simply maintaining the status quo or taking a limited approach to automation is a losing proposition. Manual operations are now extremely fragile and relatively costly; limited efforts at automation (particularly those aiming to simply maximize the number of completes per panelist) can actually have detrimental effects on everything from feasibility to reliability to data quality.

Automation allows us to process this information in real time, allows for variation in inputs (the bazillion different questions someone could ask to a gajillion different types of people under a squintillion different environmental conditions), and–most importantly–react to it. In fact, automation and programmatic algorithms are the best hope we have to improve respondent engagement and data quality. They are the only way we can possibly evaluate complex processes at scale, project outcomes, and then take action in ways that would be computationally and logistically impossible for human operators. But yet, we can still have human operators–trained and well-equipped–reach out to clients and alert them of real issues that threaten the delivery of their study and the data they are counting on to make decisions.

Third, for those who haven’t already, we need to migrate our business models to a new future.

The first two points don’t really happen without people believing that they are experiencing business-critical events. The real trick is repositioning the business, which is like changing wheels on a speeding train. This is where those who believe sample is a commodity should be paying careful attention. For if you believe sample is a commodity, you are essentially saying that you are either unimpressed by your supplier’s commitment to these issues or are otherwise unaware of what is happening in the industry. Fortunately, neither of these is a permanent condition.

Conclusion

The latest GRIT results put an even finer point on an old issue. There can be no disagreement about its gravity. But where we do ourselves a disservice (as an industry) is by imagining solutions that are rooted in the past. The very same technologies and market forces that carried us to this place are precisely those that hold the greatest promise for renewal. Suppliers already understand that this is a road that must be travelled. The faster buyers understand this, the more they will hasten a better future.