Civicom: Your project success is our number one priority

GRIT Says Panel Woes Are Jeopardizing MR’s Future.

A running theme through the 82 pages of the most recent Greenbook Research Industry Trends Report (GRIT) is that the quality of survey sample has eroded to the point of crisis for market research.


By Michael Smith

A running theme through the 82 pages of the most recent Greenbook Research Industry Trends Report (GRIT) is that the quality of survey sample has eroded to the point of crisis for market research.

GRIT sounds a repeated alarm over what its authors call “a known problem…with no solution in sight.” But there is a solution — all-mobile panels — which I’ll explore in a bit.

First, some facts and quotes from the report that lay out the dimensions of the panel crisis:

  • 38% of GRIT’s more than 2,000 industry respondents expect sample quality to get worse over the coming three years; fewer than 28% believe it will improve – and among clients of market research providers, optimism sank to 23%.
  • “Clients and suppliers agree that sample quality is getting worse, and there is little alignment on what to do about it. This is a perennial topic; when will the industry do something about it?”
  • “The smartphone revolution and declining participation are indeed problems that need to be addressed. Few disagree with this belief, but there is far less consensus around the extent of the problem, its implications and the range of solutions.”
  • “The difficulty of accessing truly representative sample sources….could be viewed as the single largest area of concern for the industry….We are running out of online panelists…”
  • “There are few legitimate excuses one can muster for not confronting the sample problems that plague the industry. There’s no doubt that the solutions are hard, but…far too many people…are dragging their feet.”
  • “The real existential threat to our industry is…the future of research participation. The real question therefore is when will people catch on? When will responses to these questions drive change?”
  • “We believe that the death spiral is accelerating for those researchers who fail to act. The poor experiences they create are starting to contrast markedly against the unique and engaging experiences by new entrants as well as the small number of innovators who’ve been unafraid to embrace change.”

The last sentence points the only way forward. Innovate. Embrace change.

A Formula for Successful Mobile Research

My argument is that state-of-the-art mobile research is the innovation our industry needs to embrace. But before that can happen, we have to overcome a common misconception about what mobile research really is, and what it can accomplish.

The misconception is borne out by one of GRIT’s most telling findings: 74% of respondents think they’re already doing mobile research, more than any other “emerging method.” An additional 17% are considering trying mobile for the first time.

MFour has long struggled to make the industry realize that not all mobile research is created equal. There’s good mobile and bad mobile, mobile that’s artless and mobile that’s state-of-the-art. There’s pure mobile that’s solely geared to smartphones, and diluted mobile that ties smartphone respondents to fading online survey technology. There’s mobile that fails and mobile that works.

MFour Mobile Research, Inc.’s aim since 2011 has been to define what mobile research can and should be, then create the new software and new approaches to panel-building that alone can make mobile work. Success means solving both ends of the equation: developing the right technology and recruiting and cultivating the right panel.

Developing The Right Mobile Technology

We’ve broken with all trappings of online research. Instead, we deploy technology that’s new to market research, the native app. Our proprietary app, Surveys on the Go®, instantly loads an entire survey into the respondent’s phone – including any pictures or multimedia content needed to enhance questions and answers. Embedding the survey into the phone is what makes it “native.”

Why does it matter? Because it frees respondents to complete surveys at their convenience. They don’t have to interrupt what they’re doing. They don’t need to be connected to the internet. Consequently, there’s no risk that the survey will become intolerably slow because of poor connections that lead to snail’s-pace downloads and data transfers. The survey can’t be dropped because of a lost signal.

At the opposite end of the spectrum are hybrid approaches that tether mobile devices to online survey software. A separate, back-and-forth exchange must take place for each and every question and answer. It’s a method that puts the respondent’s experience and the survey’s success at the mercy of internet connections that, as we all know, can bog down or disappear.

Essentially, mobile surveys embedded through a native app don’t have to be short and simplistic. Immune to smartphone signal issues, they can be long and sophisticated, and exploit special smartphone capabilities such as multimedia and Geo-location, which allows inviting panelists to surveys while they are still shopping or have just left a store. In our experience, app-based interviews run smoothly, regardless of location and even with interview lengths exceeding 20 minutes. Even at that LOI, we’ve experienced just a 6% drop-off rate. So much for the five-minute survey limit that’s commonly but wrongly posited for mobile research.

As for building a reliable, representative sample, good technology that begets a good respondent experience goes a long way toward drastically improving participation.

Curating A Winning Mobile Panel

With the right mobile technology, it’s possible to recruit the right kind of mobile panel. Ours numbers more than a million active respondents who take surveys solely on their smartphones and other mobile devices. They seem to like it, as reflected in strong ratings and comments at the App Store and Google Play. The mere fact that respondents can give us direct, unsolicited and very public feedback on their survey experiences makes app-based mobile a superior tool for becoming aware of panel problems as they arise – and taking quick action to solve them. It makes us accountable – as any firm that’s serious about its responsibilities and confident in its capabilities ought to be.

There’s much more to tell about the all-mobile approach – not least its ability to reach Millennials, Hispanics and African-Americans who, as GRIT notes, are vital to research but increasingly inaccessible to online surveys.

The Solution to Successful Mobile Research

But my main point is that the industry needs to understand that the available mobile technologies differ drastically. Then firms can make the natural comparisons, try different mobile providers, and see which can deliver a good panel and fast, reliable, representative data.

I think the most important sentence in the new edition of GRIT comes near the end, in a section called “Opportunities for the Market Research Industry” that examines ways forward from the current dead end.

“Mobile research has been seen as an opportunity for many years, but there is a sense that now we are at the stage where we can really start to exploit mobile data gathering techniques.”

Before you can exploit mobile techniques you must get to know one technique from the next. You have to stop stereotyping all iterations of mobile research as prone to the same limitations and drawbacks.

GRIT has done our field a great service by refusing to sugarcoat the sample problem and by sounding a clear alarm that something has to be done about it. There’s just one point I dispute.

I wouldn’t say that market research’s pervasive sample woes are “a known problem…with no solution in sight.” There is a solution, but until now it has been overlooked.

That appears to be changing. MFour’s year-by-year growth since we debuted our native app in 2011 suggests that an increasing number of researchers are starting to make the kinds of distinctions about mobile research that need to be made.

Market researchers need to do some research in their own backyards, to gain insights into their own most crucial interests – especially, as GRIT makes clear, when the industry’s health depends on it. Getting a more sophisticated understanding of mobile, market research’s most widely-adopted but least understood “emerging method,” would be a good start.

Please share...

2 responses to “GRIT Says Panel Woes Are Jeopardizing MR’s Future.

  1. The real question seems to be the attitude about surveys more so than the exposure or willingness to participate in a panel. The panel issue has been a struggle ever since we started collecting online responses, it’s not new. The bigger issue was the misuse of surveys and the perception or perhaps correlation of value associated with participation. When the value proposition shifted from change to reward, the survey process became muted.

    Let’s be honest and admit that relationships are needed for good data. Understanding advocacy or issue is important. Finding brand or product enthusiasts presents one perspective, finding interest that is not brand loyal is another.

    The sheer volume of exposure to products, much less surveys, has lots of navigation going on designed to avoid outreach. Heck, even those helpful hints on Amazon aren’t really that helpful and even they can’t seem to stop advertising products that you once surfed a month ago. The requests from suppliers for personal evaluation and ratings is huge. If I buy one brand I get a grocery receipt coupon from another. I suspect most people have only one thing to say to suppliers: stop. Just stop.

    So, what is the answer? One on one feedback is a good start. It’s not very efficient but then again, we don’t live in a mass marketing world. The last thing millennials want is to be aggregated. The real job of researchers isn’t to validate; it’s to explain. We don’t do much of that anymore. Instead of surveys, why not explore white space with people who don’t bucket on a particular product or service? Why not admit that people aren’t consuming media in the same way and that they aren’t getting the information they rely on from the TV or media? The internet is interactive and it presents a huge opportunity to speak with people who will relay their experiences in their own way and far faster than any survey can deliver. Quantification isn’t the answer because metrics answer most efficacy questions pretty quickly. The secret sauce is in the road less traveled.

  2. We need to distinguish large public services from smaller surveys of specific populations. With the latter, if we’re dealing with employees or B2B customers, there are ample tools already and its possible to achieve response rates of well above 50% using conventional methods.

    The large, public or consumer surveys are problematic, and it is in that context that Ms. Woods is absolutely correct. It really doesn’t matter how we try to reach respondents if only a small, biased portion of target respondents choose to respond.

    Among the public, the research industry suffers from lack of respect:

    (1) Over use (“familiarity breeds contempt”)
    (2) Survey misuse (which is still happening, especially this election year)
    (3) Lack of publicly visible, authoritative leadership that commands public respect. There is no one who commands the public awareness and respect that, for example, George Gallup Sr. or Louis Harris held in 1970.
    (4) A perception in companies that amateurs can do this and that professionals aren’t worth the cost.

    When you couple these issues with Milton Friedman’s classic “free rider” argument — I don’t need to act because someone else will and I will get the benefit of their labor — you have a perfect recipe for surveys producing imperfect results.

    Cellphone technology doesn’t fix that, unless there is some obscure reason why a majority of people who receive these surveys would complete them. Even then, if the method requires “opt in,” then the damage is already done.

    I suspect this year will bring further erosion in consumer confidence in surveys as we see inaccuracies in election forecasts.

Join the conversation

Michael Smith

Michael Smith

Chief Product Officer & Panel Director, MFour Mobile Research