SIS INTERNATIONAL RESEARCH: NYC FOCUS GROUP FACILITY
Our new GreenBook Directory site is live!
Your #1 strategic guide to consumer insights.
Qualtrics X4 The Experience Mgmt Summit
Brand & Retailer tickets for all IIeX events now start at just $99! Get or give one today!

AI’s Biggest Impact on Market Research Should Terrify Us All

AI brings with itself a lot of excitement, and additionally equal concerns with preventing fraud. With increasingly better technology market research professionals need to adapt fraud prevention methods to catch up.

Editor’s Note: Survey research is a hard craft to master. Many facets, from sample design to questionnaire design to analysis, take time and skill to master and can go wrong for many reasons. Combatting fraud makes a difficult job even tougher, and threatens the credibility of everything we do. In this post from Bill McDowell, we hear about one of the scourges of our recent times, the dreaded survey bot. Not only does he describe the problem, but he lays out some strategies for combatting the problem.


The market research & insights industry is buzzing with excitement over newly emerging and horizon applications of artificial intelligence (AI), and rightfully so.  Machine learning, when applied to open-end coding, data processing, survey programming/administration, participant sourcing, and even quantitative/qualitative interviewing is sure to have a lasting impact on helping us deliver faster, higher-quality insights at lower costs. However, arguably the biggest impact that AI has had on the insights industry to-date is one that all research practitioners and end-users should be very afraid of, and one that we’re not talking nearly enough about… survey bots.

Survey Bots

In a nutshell, survey bots are programs/algorithms that allow for automated completion of online surveys.  In most cases, bots are a means for unscrupulous programmers to rack up cash, rewards, and sweepstakes entries for participating in paid research studies without actually taking the time to share honest opinions.  On the surface, these ‘participants’ look to have qualified for and legitimately completed a survey, but the validity of their survey data is complete garbage.  These bots are becoming more prevalent and more sophisticated and should be an obsession of the insights industry to weed out.  For a fun, fear-inducing exercise, Google the term Survey Bot.  What you would hope to see in your results is a collection of market research and academic thought leadership on how to avoid and prevent such parasites.  However, what you actually see is a laundry list of available tutorials, services, and software downloads for executing your very own bots and gaming the system.

It’s certainly a scary issue, but not an insurmountable one.  Clients and suppliers need to remain vigilant and work together to minimize the impact of survey bots.  A handful of effective and relatively painless techniques to do so include:​​

1. reCAPTCHA

Those “I’m not a robot” widgets that we see on so many eCommerce and social media sites are quite easy to insert into most online surveys.  If you’re using a survey platform that doesn’t allow them, it may be time to start shopping for a new provider. ​​

2. Quality Control Questions

A great method to help control bots (as well as to keep human participants on-task and engaged) is to pepper a handful of short quality control questions throughout your survey.  Open-ended questions and multimedia (e.g., asking respondents to listen to an audio snippet and respond) tend to be more difficult for bots to overcome.

3. Rely on Sample Providers Who are Equally Obsessive About Data Quality

Any reputable online sample provider should be working on your behalf to prevent bots from making their way into your survey.  Additionally, any that do slip through the cracks should be removed and replaced at no additional charge.

Unfortunately, there is no catch-all solution to the issue of survey bots, which is why it’s important to take quality assurance measures during all study phases, from sample procurement to survey development/programming to data analysis.  As an industry, the best thing we can do is collectively admit that we have a problem and partner with one another to make sure we keep this issue at bay.

This article was originally published by Accelerant Research

Please share...

One response to “AI’s Biggest Impact on Market Research Should Terrify Us All

  1. You’re not alone. Some of us have been talking about this for years: https://greenbookblog.org/2019/03/22/online-sample-quality-4-years-later/

    Quoting: “any that do slip through the cracks should be removed and replaced at no additional charge.” Slip through the cracks. Not exactly difficult. It’s a large crack. How large? 20% minimum, gets tossed as an *upfront expectation.*

    If you’re a brand reading this, this is the reality. What you’re seeing on your end is what has been, at best, scrubbed of fraud. At worst, it passes direct to you untouched because OEs are ‘not worth it.’

    If you’re an agency, assuming your panel shop, who is paying out sub-minimum wage incentives to locate HHs with $75k+, for absurdly boring survey topics…think hard about that.

    I don’t care what marketing brochures say about data quality (“it’s really important and we take it seriously here at ___”) and neither should you. Their bots are smarter than ours. Their techniques are faster and more nimble than our ability to detect them. This is not breaking news.

    At TMRE in Vegas a couple months ago, there was a session with 10 or so roundtables, dedicated topics. I joined the table led by Imperium (a 3rd party online data validator) and I believe I was the only non-Imperium person at the table. At an mrx conference of many hundreds, I was the only attendee at the ‘data quality’ roundtable. Think about that.

    Talk to Brian Lamar, Peter Fairbrother, JD, Ron Sellers, Mary Beth Weber, Imperium… we’re around.

Join the conversation

Bill McDowell

Bill McDowell

Senior Partner, COO, Accelerant Research