Editor’s Intro: Words like “crisis” are bandied about with too much ease these days about too many topics. Hyperbole aside, the state of online sample is troubled indeed. As Schiela and Engelin discuss, and as corroborated by findings from this year’s GRIT Report, online sample is plagued by various forms of fraud, including survey bots, foreign agents, etc. The authors describe a number of useful steps to combat fraud, but it seems to me that the first step is for users of sample – research companies and clients alike – must vote with their wallets. It is a necessary step, without which, nothing substantive will be done. Beyond demanding good “sample hygiene”, the time may be ripe for more dramatic steps – blockchain anyone?
Back in 2006, an article in Ad Age told us that 50% of all survey responses come from less than 5% of the population. But that was over ten years ago and the sample industry has come a long way since… right?
Not necessarily. Fraudulent sample has been on the radar for a long time now – long enough to be pushed to the backburner repeatedly. Cheaters, repeaters and speeders are still putting the industry to the test (and now you can add robots to the list). That’s not at all to say the sample industry hasn’t been fighting hard against fraudulent sample, but the fact is, the pressure is on now more than ever for faster, cheaper and better sample. But of course, bad sample impacts the bottom line of business and so gambling on the quality of survey data is a dangerous game.
The good news is technology is on our side.
The bad news is… technology is also working against us.
We have algorithms and automated systems in place to immediately identify and remove bad respondents. But other algorithms are at work as automated survey bots to convert fake responses to real incentives.
Sample quality is vulnerable to explicit fraud (bots and foreign agents), implicit fraud (panel and sample design), and the ability to match appropriateness of sample to survey design and study objectives.
The fact is sample sourcing has a lot of holes and suppliers are far from transparent. Sample is sourced from multiple methods and channels of collection. This is due to industry-wide changes such as the rise in mobile research and communities. It’s also due to resell, river sampling and multiple ways to ‘earn.’ Many panel companies may seem to preach quality sample and superiority over ‘lesser’ companies, but the truth is nearly all panel companies borrow from one another. Meaning you might never know how all of your sample is sourced. How well do you know your sample provider?
Pre-survey screening is helpful, but loopholes are prevalent here too. Despite the high level of detail built into screening questions, professional respondents have been racking up incentives long enough to retire. There are multiple devices to take surveys on, which means multiple profiles for the same person. And now smarter social media platforms and online communities are causing high reward survey invites to go viral, putting surveys in front of respondents who are not qualified for the study.
Then once the survey is activated, cue the advanced ‘robot’ survey takers and foreign agents who take surveys for a living. Not to mention, there’s still the longstanding issues of increased distraction levels and survey length.
Best practices to combat fraudulent sample
This sounds bad, we know, but not all sample providers are allowing bad data to slip through the cracks. There are best practices and methods in place that are working.
For starters, clients and agencies should make it a point to understand exactly where their sample is coming from. Inherent bias exists and is controlled using blended sample. Most sample can be balanced demographically, but sourcing can impact the softer psycho demographics. It’s necessary to understand how different panels are built so you can use them appropriately.
Be stringent and meticulous in the pre-screening stage. Over the past five years we have seen a 2-3 times increase in the amount of respondents that we toss at the pre-screening stage. This is due to a combination of technology that is making it easier to commit fraud – join panels, qualify for surveys and even taking surveys – along with the shift to a more distracted world. People, in general, rarely focus on a single task in today’s ultra-connected world. This has been the key change in the last 10 years as surveys compete with other interests diverting full attention from the survey taking process. This increasing use of technology of course has a flip side, and it is shaping the the future of better checks and balances in sample. There’s digital fingerprinting, fraud and poor respondent automated scoring, duplicate identification and removal, and third party verification.
Finally, have no mercy for the cheaters and speeders, and the generally bad respondents. At Phoenix MI, we use machine algorithms that constantly monitor panelists and their responses in order to identify outlier patterns such as low survey scores, moving through surveys at high speed, and static selection patterns in multiple choice. Whether the behavior is deliberate or due to distraction, the results are removed immediately. Typically we eliminate 2-5% of respondents for being bots, survey farms, or fradulant respondents during real-time integrated technological checks. This is after typical industry standard checks take place, so it’s an alarming amount of potentially fraudulent sample.
What to watch for next
A lot of people believe the means of getting past these detectors are increasing at alarming rates, bots are getting more difficult to detect, and the continued push to drive down research costs is in turn driving down research quality.
So, what can we expect? We might expect to see a change in US panel rewards to eliminate out of country respondents. Companies may start standing up to those clients who keep asking for the addition of “one more” question. We need to say no to long surveys and keep mobile surveys short. And older, less common methods such as telephone might start to mix more commonly with newer methods to fill the gaps.
Our own experience with panel providers, when we approach them on the subject of fraudulent sample, has been mixed. Many panels take quality seriously and will react quickly to changing threats. But while no panel has ever “waved us off” when we discuss quality concerns, some panels have either been unable or unwilling to secure their panel assets. In this case we always remove them from our work and bid process. We agreesivly monintor quality and performance across our provider network, and take immediate steps to address quality concerns, and until the landscape radically improves across the board it’s the only way to be sure as you can be about your sample.
As an industry it’s time to really address the issue of fraudulent sample and stop cutting the conversation short with a shoulder shrug followed by “well, everyone else is doing it.”