The ultimate guide to transforming your business with business insights
Insights That Work

Step 2: Surveys Aren’t Your Only Quantitative Data Source

Rather than grow larger and larger panels with dozens of profiling questions repeated over and over, perhaps we should find new ways of identifying qualified participants. With some very basic filtering and analysis for targeting, I can reach response rates in excess of 30%.


Editor’s Note: Client-side researcher Jason Anderson of Blizzard Entertainment continues his exploration of the model for the 21st century market research company that he began in his provocative “8 Things I Would Do if I Were a Market Research Company” post, this time exploring moving beyond the survey for data. This is a must read!


By Jason Anderson

(For context, you should first go back and read my 8 Things post. Caught up? Good.)

One unfortunate consequence of do-it-yourself survey solutions is just how many surveys are out there. It’s a bit hard to quantify — vendors aren’t exactly publishing their annual survey volumes — but where there’s growth in online panel supply, there’s probably proportional growth in survey volume. And the panel suppliers have been growing indeed:

Yes, yes, there are issues like panel overlap and reselling and whatnot. This is not a precise market sizing activity. But if you just add up the claimed panel sizes from firms listed in Quirk’s, 1.12% of the current world population belongs to an online panel. Internet World Stats claims that only 2.1 billion of us are “online,” which means that 3.7% of the online population belongs to a panel. Even exercising an abundance of caution and saying that these figures are inflated by a factor of 3x, 1% of the Internet belongs to an online panel sample.

I’d argue that it’s way too much, which is why panel quality remains such an important topic. But I digress.

Rather than grow larger and larger panels with dozens of profiling questions repeated over and over, perhaps we should find new ways of identifying qualified participants. With some very basic filtering and analysis of my customer database, for example, I can reach response rates in excess of 30%. By targeting followers of relevant brands on Twitter, I can do the same in those cases where I need to reach outside my CRM system.

This approach won’t work today for all demographics, but every year it works better and better. This year’s college graduates never heard of Netscape, because the company had come and gone before they started 8th grade. Their educational experience has always included message boards and MySpace/Facebook. They use Facebook for messaging, not email; mobile apps are as natural to them as rotary phones were when I was a kid. Whether or not a computer remains relevant over the next 10 years is even up for debate; tablets have already surpassed netbooks in sales, and are gobbling up desktop PC share at an amazing pace. (Already, tablets + smartphones > PC.)

So if smartphones and tablets are projected to exceed laptops and desktops, you need to start thinking about shifting your data collection strategy away from surveys sooner rather than later. Your long-term strategy damned well better include new technologies that extend to these different platforms and user experiences; putting a tablet skin on a web-based survey experience isn’t going to be sustainable. You can’t fill out large grids of rating questions on a smartphone; you can’t use Flash on an iPad to “enhance” a vanilla survey; you can’t assume the participant has 30 minutes to spare.

So where do you get the data instead? My two (obvious) predictions:

  • Google, Twitter and Facebook. They’re in the data business, and would be happy to sell you targeted advertising space to reach the right people.
  • Mobile polling apps. We’re not going to want 50 different “panelist” apps on our mobile devices, so there will certainly be growth and consolidation in this space. Hopefully a technology platform will emerge that can be directly integrated into branded apps.

Tablets are still focused on content consumption, not creation, so the jury is out as to what breadth of apps ultimately will live on these devices. But they’re also much more powerful devices than a smartphone, which opens up many creative opportunities. Ship some iPads preloaded with the right software for in-home studies, and capture both usage and multimedia. Design your tablet-focused “surveys” as bite-sized chunks that can be consumed during a TV commercial break with just a few taps and finger slides. Just don’t take that 40-question survey with 5 open-ended questions, 20 grids, and 15 multi-selects and reformat it for 768 pixel width.

Please share...

2 responses to “Step 2: Surveys Aren’t Your Only Quantitative Data Source

  1. Yes, the 21st century research supplier will indeed leave formerly sacrosanct sampling methods, and theory, in the dust. Projectability? Reliability? Representativeness? Who cares? Microsoft sure doesn’t, based on the hack work I’ve seen out of the IEB.

  2. Jason, a great post and very forwarding thinking. Also very challenging for those of us in an industry that is focused on statistical significance and representativeness. As you note, technology is developing at an amazing rate and it is up to us to understand how to best use this technology to help our businesses make better decisions. However, this requires a certain amount of risk taking and a willingness to try things that may not work – i.e. for industry to be willing to experiment. Unfortunately, in our budget conscious climate, I see very few companies with this appetite.

Join the conversation