Unearth the power of declared data
Insights That Work
Brand & Retailer tickets for all IIeX events now start at just $99! Get or give one today!

The Black Hole of Research: Dull but Critical

Are you paying enough attention to the boring part of the research that can destroy your project?


By Ron Sellers

There’s one area of both qualitative and quantitative research that too often is a black hole.  Researchers can hand their precious project over to someone and expect the results back, but feel they don’t need to worry about what goes on in between.  Unfortunately, what you don’t know can kill you.

Well, maybe that’s a bit melodramatic – but what you don’t know certainly can kill your project.

I’m talking about The Field.  Whether it’s recruiting qualitative respondents, getting panelists to complete your survey online, gathering completed phone interviews, or any other methodology, fieldwork to many researchers is frequently the least exciting part of a project.  Vendors are often selected primarily by price, with the attitude that The Field is pretty much a commodity.  And The Field is a practice that too often is just handed over to someone else, such as a panel provider or a focus group facility, to be completed while the researcher turns his or her attention to reporting or designing a methodology for the next project.

But what goes on in the field is a critical part of your success as a researcher.  There’s very good fieldwork, and very bad, and it’s up to you to know the difference.

For instance, when you contract with a panel provider to have your questionnaire completed, do you know (or care) whether they subcontract part of the sample to another panel?  Whether they insert some of their own questions right before yours, potentially biasing your sample?  Whether they’re going to put it into the field for just one day, skimming a convenience sample off the top in order to move on to another project?  Whether they use a router, or direct invitations specifically to your study?

When you contract with a phone center, do you know whether the work will be done domestically or off-shored to a center in India or Guatemala, where the interviewers will be probing on your open-ends with English as a second or third language?  If the focus group facility tells you it’s not possible to fill your quotas, have you explored exactly why?

Keeping close tabs on what happens in the field is not optional to good research.  It should be a collaborative process, rather than something you simply hand off.  There are some pretty bad phone centers, focus group recruiters, and online panels out there.  There are also things you can learn in the field that will make a difference in your project.

Here’s one example.  Grey Matter Research conducted a mixed-modality survey about non-profit giving a couple of years ago.  In reviewing the data, we found a serious inconsistency:  people who said they had not given to any non-profits in the past 12 months often reported having given toward Haiti earthquake relief just a few minutes later in the survey.  In trying to figure out what was happening, we spoke with the phone interviewers.  They told us respondents were explaining that with the massive Haiti earthquake, they felt they were giving toward a country, so they didn’t consider that to be helping a non-profit.  Without that feedback from the field, we would not have been able to figure out a serious anomaly in the data.

Here’s another.  Years ago, we had to recruit focus groups in a market we hadn’t used before.  We selected a well-known, highly rated facility for the recruiting from our client’s customer list.  Unfortunately, they struggled to get it recruited.  Rather than simply open up the quotas and qualifications (as the facility repeatedly demanded), we dug into what was going on.  Turns out their recruiters were mostly single mothers who dialed while their kids were in school.  Since this was a consumer recruit, that meant most people were getting called during weekdays, while they were away from home.  We learned a hard lesson about the need to investigate the field practices of a new vendor before handing them a project – even when that vendor has a good reputation.

How many callbacks is the phone center making on each number?  Are they using true RDD sample, or listed sample because it’s much more productive (and therefore easier and more profitable)?  Did they randomize the list you provided before they started calling, or are they dialing alphabetically because that’s the order of the list you gave them?

How many other questionnaires has your online panel provider offered their panelists today – is yours the fifth one they will complete in the last two hours?  How complete and recent is the typical panelist profile – are you relying on outdated or partial information?  Is someone cleaning the data you get, looking for straightliners or speeders who completed a 16-minute questionnaire in six minutes?

Is the qualitative recruiter really using your screener as you wrote it, or is that just a suggestion for the recruiters?  When respondents ask them what the call is about, do they say, “Oh, we’re recruiting a focus group among people who use hand lotion regularly,” thereby telling respondents just how to qualify for the groups?  Are they just sending out an e-mail blast to their database, telling respondents they’re recruiting a group among women with post-nasal drip, or are they actually contacting and screening people in an unbiased manner?

In short, how much attention are you paying to The Field – this rather unglamorous but critically important element of your research?  If the answer is not “a lot,” your projects are at risk.

Please share...

4 responses to “The Black Hole of Research: Dull but Critical

  1. Excellent points, wholeheartedly agree. The dilemma for me is that the distance between the end client (e.g., myself) and “the field” is about the same as from here to Mars. It’s a black box, the domain of my trusted vendors. I don’t really want to get knee-deep in vetting the recruiting process, I just need to know that I can trust the data.

  2. Thank you soooo very much for that article. You are so on “the money”.
    A good field director with all their knowledge is pertinent to successful and most of all quality data.

  3. Jason, the key is that SOMEONE trusted, experienced, and in the know is fully monitoring the field. Most of Grey Matter’s clients leave the field to us – they’re trusting us to manage this part of the work. Fortunately for them, we’re pretty good at doing this. I basically mistrust data until I prove to myself that it’s trustworthy (which can drive some of our field vendors nuts). But I’d rather a few field people pull their hair out during the field process as I review the data than have the end client pulling their hair out once the project is done because something went wrong. If you’ve got trusted vendors and a good reason to trust them, then they’re the ones in charge of monitoring the field. Of course, a spot inspection here or there isn’t a bad idea, either. 🙂

Join the conversation