The Methodology That Is Overcoming Some of Market Research’s Biggest Obstacles

Editor’s Intro: Regular readers of this blog will know that we’ve had several posts lately bemoaning the state of online sample, and suggestions for what to do about it. The authors of the present article discuss an interesting new alternative they call Random Device Engagement, based on engaging with potential respondents while they are using mobile apps. Will this be enough to get more researchers to design mobile friendly (or even optimized) surveys, given some of the inherent challenges with length of survey? Time will tell.

---

The world was taken by surprise by the results of the 2016 US Presidential election, and none more so than the market research community. If an event as highly financed and heavily scrutinized as political polling could miscalculate this outcome, imagine how flawed the results must be across other market research studies relying on the same methods, but lacking the public accountability?

Radical shifts in consumer behavior coupled with new technology adoption have led to flaws in survey methodologies which plague the quality of data throughout market research; the election simply shed light on the pervasiveness of the issue. Random Digit Dialing, debatably the least problematic technique in the past several decades, has deteriorated due to the decline of landline phones and single-digit response rates tied to changes in technology, such as caller-ID and the ability to block unknown numbers. Other popular methods, such as online panel sampling and assisted crowdsourcing (inviting respondents via ads placed on social networks), are unable to provide comparable data quality as they rely on invitations to non-representative networks, rather than representative populations.

These audiences are biased in non-measurable ways that include panel fatigue, where respondents create asymmetric errors as they answer numerous questions, and panel conditioning, explained in a political example as “even if panels recruit a sample that looks like the perfect cross-section of the desired target population at the time of recruitment, the demand to answer political surveys turns these initially representative panelists into a bunch of very politically aware citizens.” This creates a challenge for today’s researchers to get qualified respondents in any kind of timely manner without sacrificing sample size, randomization, or settling for (unconsciously) biased audiences.

However, there is hope for an evolution of market research methodology to restore accuracy. Dr. David Rothschild, an economist at Microsoft Research, and Dr. Tobias Konitzer, C.S.O. and co-founder of PredictWise, were able to accurately predict the election and other political outcomes with high accuracy using a framework called Random Device Engagement (RDE) and Organic Sampling, a new sampling methodology introduced by research provider, Pollfish. This methodology maintains the organic and randomized benefits of RDD, while embracing current technology to get responses from a representative audience at scale. Rothschild and Konitzer released a whitepaper that outlines the gaps left by current methodologies, suggesting that the combination of RDE and Organic Sampling could be the future of market research.

 

Random Device Engagement

RDE is a new delivery framework, upon which Organic Sampling methodology is dependent. Mobile surveys began to surface in 2009 and have now been adopted into nearly every survey platform’s delivery strategy. RDE iterates further. By implementing a single line of code called an SDK on the back end of mobile apps, survey platforms have a simple, but instantaneous, line of communication with users who have installed apps on their devices that can deliver surveys and collect responses. Once established, RDE uses programmatic, a process traditionally used by advertisers for intent-based behavioral targeting, to target extremely narrow, yet randomized, audiences with mobile surveys. The SDK allows them to quickly gain the responses from a distributed mobile audience, solving for the randomization issues that most methodologies face.

It also solves for the issue of sample size. As of September 2017, approximately ⅔ of the world’s population were using mobile devices, and in 2018, the only thing harder to find than a landline is someone who doesn’t have an app on their phone. This delivery framework inherently reaches a massive audience, and allows researchers to connect with respondents where they are now spending more of their time—mobile apps. Comparatively, panel sampling struggles with declining participation and recruitment of representative populations. The expense of onboarding new panelists combined with a lack of transparency around sources that they buy from often spirals into issues in data quality. Because RDE is reliant on an audience of real mobile users rather than impanelled respondents, the audience is massive, distributed, and inexpensive due to cutting the onboarding costs.

This is the natural evolution of RDD, allowing researchers to tap into a mobile-first world on a global scale.

 

Sampling Methodologies for RDE

Organic Sampling

Organic Sampling occurs when a survey is delivered randomly to users already engaged in their apps as a part of their mobile experience. Respondents are given an optional invitation to participate in quick, mobile-optimized, and natively integrated surveys within the apps in which they are being served, which allows for additional benefits such as fraud prevention and user identification.

In contrast, a traditional online survey is sent to a panelist to complete at a later time, removing them from their regular routine and opening up the opportunity for panel fatigue and conditioning responses in an effort to move through the process. This does not produce organic responses and the panelists are often incentivized with a gift card or a cash reward, further encouraging them to submit more surveys rather than quality ones.

The direct partnerships with app developers are at the cornerstone of Organic Sampling. By natively integrating with manually vetted publishers to ensure the quality of the partners, companies that use organic sampling are able to capture respondents’ unique device IDs and offer a double opt-in invitation, first asking respondents if they are interested in participating, then using that device ID to create a robust profile of their user behavior and demographic data. If, at a later date, they are determined to fit the targeting criteria set, they will receive a survey as an advertising alternative— delivered randomly and programmatically through the app publishers. The respondent profile ensures that multiple accounts, or “bots” (machines emulating human behavior) are not being used for fraudulent responses.

Inorganic Sampling

There are methods of sampling that look a lot like RDE, but are inorganic. Companies leveraging this approach use the existing ad frameworks to deliver surveys as no-pressure banner ads within the apps themselves, prompting users to answer a screening question, then complete a survey that pops up. They key difference between this and organic RDE is leaving the app for the pop-up survey. While this approach allows collection of respondents’ ad IDs, it does not guarantee detection or prevention against fraud. The users are not profiled prior to delivery, and therefore open to the same methodology flaws that plagued River Sampling. There is very little known about the respondents, or their answer quality. In short, it is similar in framework to organic RDE, but the responses are not naturally occuring and there is less control of the quality of the sample.

The market research community is in need of a change, and political polling inaccuracies were the wake-up call. As data quality continues to suffer from slower responses, lacking statistical significance, and increased cost due to low incidence, market research loses its value and becomes barely more than educated guesswork.

Trends in technology and changes in consumer behavior should be embraced by market research. Random Device Engagement and Organic Sampling are paving the way there. The combination positions itself to grow the reach and participation of more consumers every day while ensuring data accuracy at scale and a low cost, making a compelling case to be the new industry standard for quantitative market research.