Civicom: Your project success is our number one priority

Why Screeners Must Go Away

Think about making meaningful connections out of all the ‘non client specific’ data that we collect from respondents, and using them to address the biggest annoyance of the panel members by eliminating the screen-outs by means of intelligent targeting.

a better way


By Pala Kuppasamy

I was sitting at the Dulles airport in Washington DC after missing my connection to attend the Market Research Innovation Summit organized by Research Now with a salmon fishing event in Alaska, when this unrelated thought struck me. Why are we screening out people so badly and madly in research studies? Especially in this era of mobile and big data where you can avoid these screeners mostly if not totally?

Someone told me sometime that the screener questions came into existence when the pen and paper studies and telephone studies migrated to online studies. As you had no clue who is responding, what gender or age the person is, and since some people in some families shared (?!) email addresses, asking these ‘screening and demographic’ questions every time became important.

But why anymore?  All big research and panel companies are profiling their members, keep these info fairly updated, and yet ask people the same questions they already have the answer for, and screen them out mid way. This is counter intuitive, euphemistically speaking.

Have you met some of these panel respondents in real life and asked them about what they think of these screeners? I have. They tell me that the screeners are the worst thing that keeps them away from taking surveys and the single most reason that disengages them with the panel.  They are less disturbed about longer surveys and smaller rewards. They do derive some satisfaction in responding to surveys. They DO NOT derive pleasure from being told that they don’t qualify : definitely not when they notice that their odds of getting qualified is becoming meager compared to the odds of getting disqualified.

This is why screeners must go away.

But what can make it go away? Its (1) Mobile and (2) Big Data.


Mobile is perhaps the most personal device that we use. iPhone and iPodtouch are the most personal devices of my wife and daughter respectively, that they wouldn’t allow me to get hold of. Until the iPhones and iPodtouch, it was their hair brushes / combs that were most personal to them.

The point is, when mobile becomes a mainstream research data collection channel, the profile information provided by the users (say via the survey app) will reliably relate to a specific individual and can be used for pre-screening them to determine if they would qualify for a study.

Combine this with the range of behavioral data that you can collect from the mobile devices – Eg., location data, app usage data, site visit data etc., you can now avoid a large set of behavior based screeners too by prescreening with these information.

Screening questions like the ones below can be avoided and by replaced with a behavioral targeting.

  • Did you fly in the last two weeks?
  • Did you dine out this weekend?
  • Were you at a movie hall in the last two weekends?
  • Do you read CNN on your mobile device?
  • Do you use banking apps on your device?
  • What kind of music do you listen to?

Big data:

Big data will be extremely useful if we make those small connections amongst them.

In the market research world, we collect various data points from the respondents and its not uncommon to ask the same questions to the same respondent multiple times. Let alone the possibility of connecting the dots and making inferences. There are various reasons why things are the way it is today in the industry, but I don’t think we should accept it as the best practice going forward. Especially in the era of big data.

In a broader context, the level of data we deal with in the research industry may not really qualify for being called as the ‘big data’, but it will get there very soon when we start to include and accumulate the behavioral / passive data we gather about respondents. There is a big opportunity to think and use this data for the respondent benefit too, besides using it to meet the client needs.

Although the term bid data has become a cliché  of late, the power of analytics – both derivative and predictive – that the big data wave is enabling, has interesting applicability to what we are doing in the market research industry – at pre and post data collection stages.

Think about making meaningful connections out of all the ‘non client specific’ data that we collect from respondents, and using them to address the biggest annoyance of the panel members by eliminating the screen-outs by means of intelligent targeting. Oh boy, we have hit on something really big, really valuable for the research industry.

Please share...

14 responses to “Why Screeners Must Go Away

  1. All very good points and certainly the point at which Market Research agencies and panel providers finally get to grips with profiling and screening their panel is long overdue. Until mobile phones reach the point that behavioural tracking apps no longer drain users batteries within hours I believe the key will be for panel companies to get smarter in regularly profiling their panel.

    n my experience some get it, some don’t and some come up with solutions which skew all their other projects in field at the same time as they fail to understand the implications of poorly implemented “big data” solutions.

    The other consideration research agencies themselves need to give is working out the expected incidence of groups they are trying to reach and if their usual methodology is right – targeting a 0.1% incidence group through an online panel probably isn’t going to work (even if you get 1,000 of the 500,000 you send your screener to straight lining through and ticking the right options!)

  2. Great – another true believer – just what we needed. I want to do a study with Mouthwash users and I need to balance the stimuli among users of Listerine, Crest Pro-Health, Store Brand, and “Other” brands – how do I do this without asking which they use?

    I’m sure there are crappy screener questions being used, but that doesn’t make screener questions bad, it makes bad screener questions bad.

    1. Great points Gregory and Steve! The behavioral tracking power issue is an issue, although by tying data into POS systems through loyalty cards or via applications like Shopkick it is certainly possible to get store and product level usage info on respondents without that complication. The secret will come through synthesizing and connecting data sets. Imagine if Google starts correlating Search, Browsing, App usage, and Mobile Wallet date to deliver surveys. They could do it pretty easily. Research Now could do the same thing by linking all that e-Rewards loyalty data with panelist social media profiles. Does that mean we won’t ever need screeners for hard to find populations? Nope, but we could target much more effectively than we do today and make research more user friendly, efficient and effective in the process.

  3. It is a lovely idea in the perfect world. Let’s imagine a panel company in this ideal world. The profiling survey is 18 million questions long (to account for all brand and behaviour and psychographic differentiators among people) and client surveys are no longer proprietary because we want to use their survey questions to add to our profiling data. All privacy is lost because panel companies gain unprotected access to your credit card and loyalty card and purchase behaviours and that’s done whether you like it or not. Ah, the perfect world. Wait. No. I don’t want to be a part of that.

  4. P.S. Pala – screener questions have been around since we started doing marketing research. Whoever told you it was an online research phenomenon should be slapped, sent back to school, avoided like the plague, or all of the above.

  5. I think we’re still a long way from when we can do away with screeners, for a few reasons.
    There’s a suitability issue: some questions can’t be answered by “Big Data” and need a survey, and some surveys are just not suitable for mobile, so for the time being we’ll need to continue with at least some traditional surveys (regardless of the channel – online, phone, in-person or something else). For traditional surveys panel sample is often OK, but it’s sometimes inappropriate to use these convenience samples. In these instances we generally won’t have profiling information on respondents, so screening is a must.
    Even when panel sample is appropriate, panel profiles don’t always have the information you need – it would be a lifelong hobby for a panelist to fill out profiling surveys with every brand and every product they use, and marrying panel data to loyalty program data to identify brands and behaviours is another whole can of worms (for instance, there’s a very good reason why ResearchNow won’t let you do a credit card survey with their AirMiles panel…).
    There are better solutions than eliminating screeners. For instance, when we do client-identified studies where we still need to screen (and we don’t want to alienate our client’s clients by terminating them) we often create a short and fun survey for those who don’t qualify for the main survey. The respondent is happy, we get some marginally useful extra data, and the cost is minimal.
    Mobile research and Big Data are both bringing a lot to the MR table. They are part of the future of MR, but they will not replace all traditional survey research. At the end of the day, we still need to be able to make good surveys, and that means finding a good way to screen respondents.

  6. One of the few questions senior marketing folk ask about Research Methodology in a debrief is: who did we talk to? Subtext: did we talk to the right people? Screening is extremely important, and often Panels pre-profiling take you so far, but then….30% (to take a figure) of the rest of the questions need to be asked. Is that “niche”? If there’s a better way to do screening, then – bring it on. But make it robust, quick, cost effective.

  7. Nice POV, Pala.

    I can definitely confirm that screening is not an online phenomenon and has been a part of the industry all the way back to sight screening in central locations and demo and interest profiling on the phone.

    However, I understand that you are not advocating dismissing screener questions altogether when they are unavoidable to get to our target, but to utilize what we should already know about consumers to minimize the intrusion (and redundancy) of our questioning.

    I’m very optimistic about the various solutions that are coming into the marketplace to give us more passive insight into consumer interests and behavior. I think this evolution of data mining will go a long way toward allowing us to garner a much better understanding of who we are talking to without having to ask a ton of annoying and repetitive questions that ultimately result in a non-rep sample of outliers who represent the small minority willing to spend 20+ minutes of their life answering inane questions for $2.

    Very impressed by the fairly recent sample provider apps that provide insight into what consumers like and follow via Social Media and the demo/psychographic background that can be gleaned as metadata (though obviously updates will be required as people progress through life stages).

    I believe the Google approach of asking one question at a time but building to a greater collective insight is eventually where we will evolve (via mobile). However, it probably won’t happen while most of us are still driving the industry’s direction …

    Some people are willing to give us a window into their lives with very little censorship of behavior. This seems to be increasingly true with the younger generation that has adopted Social Networking and Geolocation check-ins as a normal way of life and do not appear to be hindered by the privacy concerns of their progenitors.

    With opted-in consumers, I now have the ability to know what they have viewed on TV/online and what transactions they make. With Google Glasses and the like, I may even know what they saw, picked up and purchased or rejected at a store. This is very powerful information and can help us make some insightful marketing and positioning recommendations.

    At the same time, we need to do some more research about how those who are so transparent about their behavior may be vs. the “typical” consumer in order for us to make the most effective recommendations as marketing consultants.

    It is truly a new era in the industry and we need to leverage all of the information we can while making our consumer interactions as pleasant and honest as possible. This will enhance our abilty to make our conclusions based on real and heart-felt data rather than doing our best interpretation of data soup from fatigued respondents …

    In the end, someone will ultimatley offer a DNA analysis of our sample that will provide more info than we could ever imagine. In the interim, we’ll keep taking baby steps toward knowing a little more about our respondents without having to probe. Shorter surveys, more insight, more focus on solving the business question. Eventually we will get there …

  8. Greg, Steve, Scott, Rob, Lenny, Annie

    Great comments. Thank you.
    As you can imagine, I dramatized the subject line / topic to make a point. I don’t expect screeners to go away in total. Its the bad screeners that must go away.
    Sometimes connecting the dots and the data points we have in hand can provide a huge life. Take the listerine example here. We are testing with a weekly tracker that asks our members to scan the basket of goods that they purchased every weekend, with their phone and the survey app. That info will tell us who is buying what.
    The point Im trying to make is that mobile and big data analytics are empowering us more and more to get rid of most of the screeners and this is the time to look to reinvent ourselves!

  9. If you have a behavioral tracking app that gathers all the answers you mentioned (dined out, flew, music use etc….) AND Apple will allow it into the AppStore you will have every MR company (plus the NSA) beating a path to your (virtual) door. It will also be interesting to see how much panelists will ask you to pay them to give up that level of privacy.

    I totally agree that screening out respondents with “I’m sorry but you don’t qualify!” particularly after 10minutes of detailed questions, is completely unacceptable. Normally you want to screen out fast and take them to a different survey if they don’t qualify for the first one. Most panel companies already sample based on known profile information but sometimes you need an in-survey question because you didn’t have it in your catalog before and you’re in a hurry. Or it’s a one-shot question for one project and it’s just not worth adding it to your catalog.

  10. This topic has been around for 10 years. It’s amusing to see it arise again with the belief that mobile and big data solve the issue. Everyone agrees that we need to get rid of 10 minute screening processes that end with “sorry you don’t quality”, but why don’t they go away then?

    First, you can’t just collect 100% of all data from everyone. As we all know, panelists take surveys for a reward and the CPIs don’t support a full reward mechanism for 100% of your sample. Second, panel companies take their orders from market research firms. While we can say, “no, I won’t conduct that survey against my panel with that screener intact”, there’s certainly a company that will and then we all lose our formerly loyal customer to a company with less concern bout panel quality. The “big data” concept of storing tons of data I specialized data warehouses doesn’t solve the expectation market researchers have of standardized, structured pre-screening data collection.

    As for mobile, the idea that behavioral targeting can solve screening whilst searching for needle in the haystack sample is naive. I’m a huge proponent of mobile, but the scale of behavioral data collection on mobile is meager today and makes it impossible to fill quotas. Screening questions don’t soley revolve around location, app usage, and music preferences. When mobile behavioral targeting can tell me what brand of toothpaste you use, without me asking you, then we’ll be onto something.

    But I do agree that bit size data collection, ala Google, is a step in the right direction. Yahoo began progressively profiling in the ’90s when they asked you a single question every time you logged in to check your email. But I still don’t think Google or Yahoo know that I prefer Crest. But I guess as of right now, they do.

  11. Three years back people thought it was naive to think that mobile can be used to run meaningful surveys. Still many think it is only conceptual to collect behavioral data. But things are changing at rapid pace. If we keep our eyes wide open to whats going on in the broader technology space and apply the relevant ones creatively to MR space, lot of what we think is not possible will be possible. Farmers once thought that grapes harvesting can NEVER be done by machines.

  12. Screeners aren’t the issue. Poorly designed Screeners and poorly handled recruiting are the issue.

    RE Screeners — don’t make them the lengthy. Only ask the essential questions in a manner which does not to tip your hand. We need the right people in the research, so we must screen for them. But sometimes the screening becomes excessive fine toothed.

    RE Recruiting — When someone doesn’t qualify, the right thing to do is thank them for their time and say that that those are all the questions for them at this time. DON’T say you didn’t qualify. This makes people feel rejected. If need be, explain the need to include people with different backgrounds/demographic or product use tendencies in the research, explain the quota for their group has already been filled, and tell them you APPRECIATE their interest and willingness to participate and will reach out to them in the future.

    Lastly, mobile is great for some targeting but not for all audiences or purposes.

  13. Pingback: bola tangkas

Join the conversation