Download the
Join VisionCritical and Forrester's webinar exploring the playbook of insight-driven business on April 4

The 7 BS Stories of the Market Research Business


Editor’s Note: Client-side researcher Jason Anderson continues his quest to create a “No BS Zone” within the research blogosphere by speaking out on the reality of some of the biggest issues impacting the business of market research. Although his context is within the gaming industry, I suspect his experience translates across most categories. As a supplier-side researcher, it’s both refreshing and a bit intimidating to have our dirty laundry aired this way, but it is my belief that you can’t fix a problem unless you understand the cause and in this post Jason gets to some of the root business model issues impacting MR. This one is sure to spur some debate and I’m looking forward to hearing what you think! Make sure to check out the comments section to follow along and contribute your own thoughts.

By Jason Anderson

Last month, I saw a presentation from VGMarket (a boutique research shop in the video games space). If you hit up their website, you’re going to see something rarely revealed by a supplier: their prices. Yes, for a firm quote that includes all of the idiosyncrasies of your particular study you’ll need to talk with someone, but if you’re just trying to size up a hypothetical project you can avoid salespersons entirely.

You’re also going to notice that their prices are…well, they’re cheap, by industry standards. I rarely get quotes for gamer-targeted focus groups for $3k/piece. But this isn’t an article about a specific vendor per se — it’s about the seven “truths” they expressed about games research practices.

I agree with all of them. They’ve given me permission to post and comment upon their presentation, so here are 7 distortions that too many companies accept at face value.

BS Story #1: Focus group pricing is fair.

Actually, most of the time it’s not. If a standard focus group or playtesting group is 6 to 8 respondents for 2 1/2 hours, a typical invoice for a US-based group adds up to $6k to $10k per group. In reality, the recruiting is typically outsourced to another firm for $80 to $120 per respondent. Add on some fixed costs for facility, and that’s lots of healthy profit margin.

If you are conducting at least 4 or 5 standard groups, there is no financial justification for a vendor to charge more than $3k to $5k per group, including expenses. I recently hired a quality firm to conduct 8 groups and 20 individual interviews for under $30k. And I got great quality output. Quotes for the same service from other vendors were as much as twice as much.

Lesson: when is the last time you looked underneath the hood of your vendors’ financial model? Are you paying an appropriate margin for the value added? If you’re paying $60k when you could be paying $30k, what is the extra value delivered for that extra $30k?

BS Story #2: You can always trust that your focus group respondents meet the criteria

While I didn’t need a vendor to tell me this for me to know it to be true, it’s refreshing to hear. Most of the time you’ll work with your vendor to develop a screening survey, and you probably rely on that vendor to manage the integrity of the recruiting process.

Most vendors don’t manage the recruit directly, however. They outsource to another firm and use the screening survey as the delivery contract. Those recruiters (now two steps removed from the project) won’t have industry-specific experience; they will have survey script-reading experience.

This is particularly challenging in the games business. Not surprisingly, gamers are  great at “gaming” the system and figuring out the right answers to qualify (and earn their money). Weeding out the liars requires in-depth knowledge of games and game genres, and the ability to challenge respondents to prove their claims. For example, if your recruiter isn’t a gamer, and their target is people who have “played required game X for at least 5 hours,” how can you validate their gaming experience?

You can’t. And if your vendor isn’t paying attention to these details (either by doing their own recruiting or micromanaging their subcontractors), you get participants with the wrong background. Which leads me again to wonder why I’m paying twice as much for half the value.

Lesson: Contract management 101. Make sure ownership of recruiting process and quality control is clearly defined and there is explicit accountability.

BS Story #3: You can always trust your online survey sample and its pricing.

VGMarket’s presentation captures the problem succinctly: “Because the end customer has zero visibility into online recruiting practices, the online sample business has devolved into a cesspool of incestuous outsourcing, excessive mark-ups, and rampant fraud.” Those aren’t client-side words, they’re supplier-side.

Why are things this way?

  • Repeated outsourcing to sub-agencies and sub-sub-agencies results in costs of $3 to $5 per respondent at the bottom of the food chain, which gets marked up 4x or more by the time the invoice reaches the client.
  • Respondents belong to multiple panels, so it’s not uncommon for the same person to take the same survey multiple times as it’s distributed through multiple channels. (Before suppliers start countering with the importance of having solid panel management, I can say that I have witnessed this duplicate-response issue in nearly every panel-based survey I field.)
  • “Survey farms,” primarily located in China, Russia, and Brazil, will deliver fraudulent completes for $1 per respondent for any recruiting criteria.

Lesson: You need to know exactly where your sample comes from. And your vendor needs to be transparent about the sourcing.

BS Story #4: Market research requires a lot of time

While some types of research require more time for analysis and processing, focus groups and crosstabbed surveys do not. For me, project turn-around time is a primary factor in whether or not to “do it myself.” If I have good internal attendance at a playtesting or focus group session, I have 48 hours (best case) before the other observers in the room begin distributing their own notes and findings, and I lose control of the message.

In these situations, I need topline reports in 24 hours. I need final reports in 4 days. My business moves fast, and I presume this is true in many other industries.

Lesson: Vendors supporting fast-moving industries need to be able to keep up with the pace of their clients’ businesses.

BS Story #5: “Business Hours”

Game development is notorious for “crunch time” — that point in the project where deadlines and the 40-hour workweek fall painfully out of sync with each other. In these situations, 8-hour days become 14-hour marathons and weekends happen once or twice per month. If I’m working that hard, so should my vendor. Just in the past month, I’ve had to:

  • Repair an 18-country email campaign from a hotel room the morning of my brother’s wedding;
  • Manage two surveys from a Blackberry at 5am the morning of my TMRE presentation;
  • Edit 200MB of focus group videos from a Dropbox while en route to the airport;
  • Fix audio wiring problems in my playtesting lab 30 minutes before start time;
  • Go from “we need to do Project X” to “Project X is finished and decisions made” in 5 days;

This isn’t about “oh, woe is me” — this is just the way things are in games. I’m 24×7 on call. I need to be able to contact my vendors at 1am on a Saturday and get the same level of service that I would during “working hours.”

Lesson: Your vendor’s working hours should match your own. If you’re in a 24×7 role, you need 24×7 support.

BS Story #6: Research for games is the same as research for other products.

I’m not disparaging research in other categories. All industries have their own vocabulary and norms. Games are extremely sensitive to the lexicon that has evolved over several decades, however, which makes it difficult (if not impossible) to moderate a discussion with only a superficial understanding of the category.

Gamers have invented hundreds of words and acronyms: LFG, GG, LB, RT, UDUDLRLRBAStart, pwn, 1337, raid, crit, epic, headshot, metacritic, FPS, RTS, MMO, RP, frag, ping, PK, mule, 4X, easter eggs, boss kill. It’s endless. So when your participant says “I don’t like playing RP because it takes too long to find a PUG on the LFG channel,” can you translate?

Lesson: You need to work with people that understand your product category. This is easier for some categories than others. In the case of games research, it’s not good enough to have one analyst or moderator that happens to be “a gamer” — they need solid experience in your particular genre.

BS Story #7: It’s OK to make big decisions based on small sample sizes.

Because game development can be so fast-paced, product teams are easily tempted to use the results of a few focus groups or playtesting sessions to make large decisions. The reason: they assume it will take too long to achieve a larger sample, or it will be too expensive, and so it’s better to act with some feedback than none at all.

I suspect this issue may be more common in the entertainment industry than in packaged goods or other products and services with longer shelf life. But it’s still possible to recruit 80 or 100 respondents at a reasonable cost and deliver both quantitative and qualitative feedback. Yes, your confidence level will be lower, but I believe it would be better to do a smaller number of somewhat significant studies than a larger number of purely directional interviews.

Lesson: Make sure your internal client knows how a study’s results can and cannot be applied.

Please share...

20 responses to “The 7 BS Stories of the Market Research Business

  1. I just find a company Informatics Outsourcing from Google search. They are providing Quantitative Market Research and Qualitative Market Research with affordable price.

  2. I had to read this twice before I could believe what was in it. Well done Jason and Leonard I thought the content was honest, straightforward and completely lacking in amy BS of its own. Excellent article.

  3. An entertaining and controversial piece as usual. Can I just suggest BS #8, though: “All these problems are solvable at the same price you’re currently paying”? 😉

  4. Context is everything. Gaming context versus CPG, and deep versus quick research. The comments about pricing and analysis time relate to a type of research, quick research. If the researcher intendeds to utilise an anytical process to the material gathered in the focus group, e.g. grounded theory, semiotics, or discourse analysis, then the time taken will be longer and the costs will be higher.

    In terms of small sample sizes, they can be OK for some decisions, but not for others. However, 80-100 interviews is a small sample for mission critical quant research, the answer is 20% like it, plus or minus 10% is not great for an importnat decision.

  5. I agree with some of the content in this article, but the info on FG pricing is very misleading.

    1. Yes, the facility usually does the recruiting and that can be anywhere from $50 to $200 per head–depending on screeners, geography, etc. BUT the researcher also needs time to supervise the recruiting. I have done this myself for 100s of projects, and some supervision is required (“The guy says he is a VP of IT but he has never heard of Oracle? Ummm….pass!”). Other costs neglected in the article: incentives ($50-250 per head), travel for moderator, reporting, video montages, client food (I have had clients order in $200 meals), transcripts (can be $200-500 per group depending on complexity). And facility costs are not fixed–they usually charge per group, not per day.

    2.A great moderator costs real money. And moderators that have rich industry-specific contextual knowledge cost more. If you want a moderator who will charge $500 per group just for moderation, go for it. In some cases, that may be all you need. But in my experience, great moderators charge at least $1500 per group to moderate–and they are worth it. Do you want someone who will read a script, or someone who can manage a group dynamic, probe on the fly, and keep the energy high? Moderation requires real skill to do well.

  6. What an eye-opening and controversial post. The comment thread is highly interesting!

    Regarding #3 Online Sample, you are right. This is a HUGE problem in the industry, and has been a hot topic since Kim from P&G first spoke out about it in 2006. As a response to the issue MarketTools created TrueSample, a technology that is used to ensure respondents are Real, Unique and Engaged. The solution is actively used by many panel providers and more importantly, the solution is industry driven. A quality council that goes beyond clients into trade organizations and foundations convenes to provide direction on best practices for dealing with threats to panel quality. Quality sample IS available from a great many panel providers. Be picky!

    And, on the subject of quality – I would like to pose a question about #5 – Business Hours. Having been a software PM, I know that the quality of code is highly correlated to how well rested the engineer is. Shouldn’t the highly mathematical, creative research analysts operate the same way?

  7. I do believe that the gaming industry is a unique market unto itself so these views need to read with that limitation in mind. I think he is on target for that market (only one project experience for me), but it really does not cross into other markets such as medicine and money where decisions are more thought out.

  8. Great post – thanks again for your insights Jason. Always refreshing to read.
    To Ray’s point above, context is everything – especially when it comes to qualitative design and analysis. For example, if a series of groups include, say, a directed metaphor elicitation exercise and a projective exercise, this will call for a more vertical, contemplative approach to the analysis – grounded theory, semiotics, etc. I think you’re right: quick qualitative “check ins” do not need deeper analytical approaches, and interpretation can remain fairly horizontal – requiring less time and less money – but again, context is everything, and in my experience each project requires a different approach and pricing structure. This also includes, as Kathryn says above, incentives, travel, video montages as needed, catering if needed, etc.
    For recruiting, I am very picky about who I work with and subcontract out to. I have fostered long term working relationships with a couple of recruiters, and in one case have been working with a “go to” recruiter for almost 7 years. You’re right that “ownership of the recruiting process” and accountability are crucial to the success of a project. Towards this, I have made it a policy to have a project kick-off meeting with the recruiting manager so we an we can walk through the purpose and objectives of the project, point by point. In some cases, the client has even been in that meeting. No question that working with a great recruiter can be expensive, but in the end clients think it’s worth it; after all, making sure the right people are in the room is a keystone of solid qualitative research that can make a difference in a client’s business.

  9. I am responding to Gordon MacKay’s response-It has been my experience that if you give it away free, it will not be appreciated. and would be ignored.Don’t know why, just is….

  10. I think this is a pretty straight-forward article, but it may not be that “telling”. Kathryn brings up some relevant counter-points and as a research consultant, I will say that in effort to be edgey and controversial this does fairly discount the value of what good research partners bring to the table.

    When you go to the mechanic he/she charges for the parts and the labor that went into the project. It’s fair because their expertise and knowledge carries some value above and beyond what it may “cost” them personally. It’s a business. After all, that popcorn at the movie, doesn’t really cost 8 bucks either. If you feel like you’re being shafted then you compare alternatives and make a different choice. But, I hesitate to believe all MR suppliers are artificially inflating the industry costs, especially in this economy (I hate using that qualification) 🙂

    In terms of the recruiting and sample costs, you’re also a little light. I can’t remember the last time I have had recruiting costs come in lower than $100, and generally they are over $140. So, that can easily account for that floating $1000 that is in debate.

    Additionally, in terms of time costs, a good research partner should not increase the time that is needed from the client. If we aren’t making your job easier, then we’re doing it wrong 🙂 At the end of the day we strive to make sure that our clients always feel that fair value has been obtained and never attempt to price hike. That’s Busch League IMHO. I guess I feel like even slightly encouraging folks to consider making decisions that affect your business, large or small, purely based on who’s the lowest cost. Junk in…junk out.

    The acronyms in any industry are always a blunder, so I am not quite sure if that is unique to the gaming industry, especially having started my research life in the Pharma/healthcare world where nomenclature is king. Isn’t the real truth that good research is good research – no matter what the vertical. The research supplier should have a solid understanding of the business and the industry, but it should be pretty easy for a quality researcher to get up-to-speed on the jargon quickly. After all….we are researchers. Perhaps the more important thing to focus on when evaluating partners is the question of does this company understand our goals, issues and objectives of this project, and can the perform and deliver on their promises?

    Overall, I really do agree with the high-level construct of your points, and that there should be credibility and accountability on BOTH sides of the coin, but I fear that in efforts to have impact, some of the points are clouded by the specifics of prices/costs/and possibly unfair generalizations.

    I’m not bashing you or questioning your main objective, but I am standing up for a large group of professional individuals, colleagues and friends who I know do their best to help their clients, are passionate about their work, and maintain fair/just business practices.

  11. The growing ubiquity of broadband and webcams is enabling a new methodology that addresses many of the issues identified in this post (e.g.: cost, speed, respondent quality).

    Although not the same as a focus group, or an online focus group, or an IDI, but with elements of all three, online Video-in-Video based research significantly lowers the barriers to participant recruitment (they dont leave their home or workplace and can schedule on their own time), thus saving cost and time.

    A combined and synchronized view of what they see and do and listen to on a computer screen, with a webcam audiovisual of them, their surroundings and what they say, can now be quite easily set up and delivered, within days, or even hours, at significant cost savings.

    The combination of low barriers to participation, digital platforms and social media platforms for recruitment, and the ability to “see” (and QA) respondents, allows for higher respondent quality.

  12. Dear Jason:

    You ask “Are you paying an appropriate margin for the value added?” when it comes to focus groups. I’d be interested in knowing what you think “appropriate” is, and what you think the “value” is that you should receive for that margin.

    For the sake of argument, if a typical hypothetical CPG group costs $3K (including recruitment of 10 to seat 8, incentives, room rental, DVD recording, respondent food, a couple of client meals) and you want to pay $5K, what ‘value’ do you expect to get from that 40% margin?

    Thanks for an interesting post.

  13. Point by point:

    #1: Pricing. I’m sure prices vary widely. And I agree that a cost per interview should go down as the size of the study goes up. BUT, keep in mind that you hire talent and experience and availability. A focus group moderator can only sell their product once, can only work on so many projects at a time, and is available to you without the cost of an in-house researcher (no HR hassles, no benefits, no office, no reviews to prepare, no whining to cope with). To stay in business, a researcher needs to make enough to keep their doors open, phone lines and internet up, pay the bills and put food on the table. You get a quality resource only when you need it.

    #2: Quality of respondents: I agree that quailty of respondens is the Achilles heel of high value qualitative research. And this goes back to point #1. Have you looked at how long your screeners are these days? Sometimes you are searching for a needle in a haystack and this costs money.

    #3: Not on the issue of pricing, but on quality: Perhaps my age is showing, but I echo without reservation the concerns expressed here about online sample, and I’ll add to that … how engaged are they, REALLY, in your project? Hey, this train has left the station, but just ask questions about this, and be informed about how respondents are vetted and how some minimally acceptable level of engagement is necessary to earn an incentive. At least when they come to a facility, you can see them and judge for yourself. Not so easy in the online environment.

    #4: Speed. There is an old adage in research: you can have it “good, cheap or fast, pick any two.” Yes, things move at the speed of light today, and no, no one ever waited for research, but sometimes QUALITY ANALYSIS requires time for reflection, for connecting the dots.

    #5: No, there is no such thing as business hours in the research field, especially when you are the supplier. Get used to it. But manage it by not biting off more than you can chew. You don’t do yourself or your clients any favors by not leaving enough “s— happens” time. I’ve worked my share of weekends, holidays, over family vacations, but make sure your research partner knows how much they can handle and when it’s too much.

    #6: I have deep experience in gaming research myself, so I feel very qualified to respond here. No, research for gaming is not like research for other categories. But you might as well say that about ANY industry. What about health care, pharma, IT, social media … each industry requires more than superficial knowledge. So what’s your point?

    #7: It’s NOT OK to make business decisions based on small samples, but you know what? It was ever thus. You can be the best advocate for process in the world, and if your client makes a decision to the contrary, well, they have their reasons.

  14. A thought-provoking article with a point of view — always a great read — as are the excellent comments made in response.

    Jason, one of the things you have me thinking about is how your supplier is able to offer focus groups at this price. I’m guessing that there are enough common elements in the approach (screener, recruiting, discussion guide, objectives) that some of the design time is minimized, or can perhaps be handled effectively by a more junior person. I also wondered if the researcher can do this work for companies that are competitors — or do they just work for you?

    I do a lot of work in financial services, and other services industries, and there is a lot to be gained (by my clients) from the expertise I have and the foundational knowledge I take into every new project. However, I also see some major differences from the video gaming industry as you have described it.

    It is actually relatively rare for me to have projects that have very similar designs or recruiting specs. In fact, products in the same category don’t even come up that often. If I look at the last year’s projects, they included a lot of fuzzy-front-end exploratory work to help clients develop knowledge products or new customer-facing technology that will have a build time of 18 months to 4 years. (Let me add that it is a great joy to me that we are trusted with such interesting and exciting work, I’m not complaining!)

    As Ben and Kathryn mentioned above, rarely would I be paying for recruiting or incentives in the ranges you specify. There’s a variety of reasons, ranging from the need to use customer lists (harder to recruit), the lack of fun factor in the topic, the personal disclosure requested, or the nature of the respondents themselves (especially B2B, rapidly becoming a recruiting nightmare IMHO).

    I do think it is incumbent upon us to figure out ways to save clients money through clever design, and I have tried a lot of ways to get there. What does not work, in my experience, is trying to force down the cost of recruiting and then paying the price in poor respondents.

    I wrote an article not long ago about how to make a qualitative budget go farther. I’ll add the link when I find it.

    Thanks again for sparking an interesting conversation!

  15. This really does cut to the core Jason. I was a career market researcher for 15 years, half client side and have supplier. I’ve been doing market research recruiting for about 3 year now and I work with lots of market research supplier clients, so I speak with supplier side career professionals daily. I really enjoyed your piece thoroughly, but all I could think of was the supplier-side practical reality of that business model. I looked up VGMarket and they seem to have about 15-20 full timers, which I expected. It will be interesting to see how/if they can scale the current business model and value proposition.

    Based on my conversations with industry practitioners (which is what I spend all my time doing these days) work/life balance and base salaries are already an issue with the really talented researchers, especially the ones capable delivering the things you mention. Talented researchers are already making pretty good money. At that point, people are often highly focused on work/balance and not willing to do some of the things you currently do (and expect), or they would want a lot more money to do them. Kudos to VGMarket for working it out and creating a profitable and viable business model.

    Lenny and I have chatted about this, and the market research industry is definitely changing, whether or not major market firms like it. The top 200 firms net ~$9B in annual revs, so you’d think they would be up for the challenge. One issue is that the smaller firms like VGMarket often get bought out and merged, and their service structure and philosophy eventually changes, no matter what the acquiring company says their intentions are. And hen you are off searching for the next VGMarket.

  16. BS Story #5 – Business Hours seems to reflect a top-down, commodity view of market research as opposed to a partnership. A partnership with a vendor could help you manage deadlines more proactively, so that timing is less likely to fall apart. There seem to be quality issues as well, so that the task is done right the first time instead of being fixed at the last minute.
    Service is important and vendors are willing to share the pain for the success of all, but I have to ask this: What were you drinking at 11 PM before that call at 1 AM?

  17. Jason, a lot of what you say makes sense but I couldn’t disagree more with BS Story #6. I’m on the supplier side and I work long hours and weekends too when it’s necessary to serve my clients. My cell phone is always on and they can always reach me if something critical comes up. Maybe I just have great clients but they would never think of calling me at 1 AM on Saturday. They realize that everyone, has another life and, believe it or not, there are more important things in life than market research. It sounds like it’s different in your industry. I’m sure it pays the rent; but remember this, you’re not curing cancer or waging a war. If so, you might have an excuse for demanding this of your suppliers. You’re marketing games for God’s sake! How important is this, really, compared to all the other issues in the world. I don’t know if you have a wife and kids but, if you do, I wonder what they think of your 24/7 work week.

Join the conversation