PureSpectrum - Schedule A Demo
Our new GreenBook Directory site is live!
COVID-19 guidance, tips, analysis - access full coverage here

Are DIY Platforms Ready for Prime Time?



By JD Deitch, Ph.D.

In 2013 I attended the CASRO Online Research Conference (now CASRO Digital) in San Francisco. One of the sessions that caught my eye was a roundtable discussion on do-it-yourself (DIY) survey tools. The program notes set the stage as follows:

“Much has been written about the recent proliferation of companies offering DIY/Instant Research methodologies, which are typically inexpensive. Are these methodologies able to produce representative results? What conditions are necessary to use such offerings effectively?

Representing the traditional industry on the panel were two respected and well-known researchers. The two biggest DIY firms (then and now) were at the table represented by senior officers.

The typical firmographic composition of a CASRO event meant that the DIYers would be heavily outnumbered and ensured skepticism from the off. It was therefore hardly surprising that the discussion never got beyond topline rhetoric. The researchers (on the panel and in the crowd) observed that the existing DIY solutions didn’t offer quality controls or oversight that one got from full service companies, which thus intrinsically and greatly increased the likelihood that people would do bad research. That these tools were being deliberately targeted at nonresearchers was, to some, like giving kids booze and cigarettes; it was morally reprehensible and threatened to bring the whole industry into disrepute. The DIYers rather unapologetically replied that people were already making choices, and that these were neither spendthrifts nor the feeble-minded but rather some of the traditional industry’s best and most astute clients. There were a few questions from the audience that tried to strike more thoughtful chords, but there was little room for common ground.

Two very momentous years have passed. What used to be a tiny insurgency has become a viable and consequential sector of the market. There are those who would view commercial success as the surest sign that self-service tools are ready for prime time. Are they though? Are they good enough? The answer is an emphatic yes.

Their tools are complete, integrated, automated, and fast.

While there’s variation in both form and function across the space, DIY solutions are built from the ground-up to be maximally automated and fast. The best are horizontally-integrated and offer an all-in-one package of questionnaire programming, sample supply, data processing, and reporting that just leaves the industry’s traditional solutions in the dust. They are cheaper to run and don’t require days or weeks to produce ugly PowerPoints. They are indisputably better.

Their question formats are better.

There. I said it. With much of the industry desperately late in transitioning to a mobile-first, device agnostic world, the respondent experience for most DIY tools is downright refreshing. Many of these companies started as technology plays and thus understood implicitly that mobile friendliness was nonnegotiable. With no installed base of trackers to worry about or existing factories that needed to be retooled, they faced neither of the major obstacles that traditional firms did/do. Most have spent time on user experience and understand its implications for data collection. Some go so far as to hard-code character and response option limits so as to prevent the sort of questionnaire design problems that even the traditional industry seems unable to regulate with its clients. Even the interfaces for the survey creator are simple and fluid. Are legacy firms making progress? Yes, their tools are getting better, but their execution remains mired in the massive number of projects that are still desktop only.

They are tackling and solving sampling problems.

Yes, you read that correctly. DIY companies are leading the way into programmatic sample, and in doing so taking out needless cost, enabling greater diversity, and still using reputable sources. That the questionnaires are short, well designed, and device agnostic means even broader coverage of audiences that are abandoning the traditional industry’s studies and panels. Some have their own panels, which they’re protecting precisely by not fielding the long, boring, desktop-only surveys that plague the legacy industry.

They are educating…

Whether they’ve come to the conclusion for commercial reasons or out of a belief that they need better quality (probably both), DIY companies are acquiring research competence and putting it to work. They’re hiring people who know research. They’re offering pre-packaged question modules and sample design recommendations to address the areas of greatest risk.

…and are attracting conscientious clients.

Amongst the researchers at the CASRO conference, the prevailing wisdom was that the potential for harm was great, especially for nonresearchers. Implied in this concern was the belief that (a) DIY firms were going to prioritize business growth over quality and (b) their clients were lemmings blithely jumping off the cliff. My conversations with people across the industry—with clients, DIY firms and observers—suggest that this hasn’t happened. To the contrary, it appears that clients are not only conscious of issues, many are asking for help. This is driving DIY firms to adapt and offer consultation.

Is there still a quality risk?

Are there people still doing bad surveys on DIY platforms? For sure, the answer is yes. But two things are abundantly clear. First, either by design or through deliberate action, DIY companies are taking clear steps to ensure quality while retaining their attractive price and speed. Second, they have resolved the quality problems that continue to vex legacy research firms, namely the glacial transition of long, punishing desktop-only surveys that are stressing panels and failing to cover a large swath of very desirable respondents. Put differently, there are plenty of quality risks that remain in working with traditional firms.

All grown up

In my original article on disruption in the insights industry, I stated that the winners of the revolution that continues to sweep through the sector would have a few central characteristics. They would make their clients’ money by getting closer to their execution. They’d build new assets. They would breed quality by design. They would ooze efficiency. Finally, as a result of these factors, they’d be able to create a new model to achieve profitability.

As the saying goes, I now have a horse in this race. After many years in the traditional industry, I’ve joined one of these DIY companies because I think this is the future of research. Don’t take my word for it, though. See it with your own eyes. DIY firms are solving problems that research agencies are not, or are slow to solve. They’re starting to change the way people think about the impossible “research triangle” where, given the essential components of quality, speed, and cost, optimization is supposedly only possible across two. They face their own challenges about building and sustaining business, but they’re fundamentally doing what Clayton Christenson said in his seminal article about disruption. Because they’re not locked into legacy business models that militate against big change, they’re building the next generation and, in turn, leapfrogging the incumbents. They—we—are here to stay.

Please share...

13 responses to “Are DIY Platforms Ready for Prime Time?

  1. Thanks for the great post, JD. It’s fascinating to see DIY research technologies proliferate and be adopted by all types of users–from established firms to small, scrappy marketing teams to local businesses trying to better understand their customers. Much like big data has created a world in which we are all data conversant, I suspect these tools will begin to make us all research conversant, too.

  2. One has to be careful when making value judgement about business tools that are careering down the track with amazing success, especially when faster and cheaper are the underlying mantras. There is no doubt this trend will continue and research dinosaurs obviously need to get out the way to some extent.

    However missing form all this is one big elephant in the room, which will plague a lot of DIY work. I know it because I have seen the utter rubbish generated by MBA’s working with major consulting firms. It’s this! The DIY researcher is plagued by one major disease – the belief that being able to ask a question will generate a sensible answer.

    This is undoubtedly what marks the skilled quantitative questionnaire designer from these inexperienced “newbies”. Its not something one could ever learn without making mistakes or sitting at the feet of people who do know how to ask questions, and then for many years. No amount of democratizing the research process with “instant DIY” solutions will solve this problem.

  3. Great post JD.

    I think one huge issue the traditional MR community ignores is budget. There are a huge number of companies who may only have $50k or less budgeted for MR costs. They may have a person but not much else. This means the DIY tools are the only way they can carry out research. The idea that they should not do research just because they have a low budget is absurd. Obviously for these (legion) companies DIY is the only way to go. It’s not the case, in my experience, that the researchers in these company are always unskilled, they just don’t have budget. Rather than decrying DIY tools the traditional researchers should be working on ways to provide services to this market place, because it is growing fast.

    @Chris – to your comment : “The DIY researcher is plagued by one major disease – the belief that being able to ask a question will generate a sensible answer.” I think this is a disease of all MR – DIY or not. 😉

  4. One of my former employers began developing a questionnaire design tool in the late 80s and it was deployed in or about 1990. Over time it was updated and extended and I found it a great productivity tool and it cut down considerably on clerical work and left more time for thinking. This sort of tool, if it includes professionally-developed banks of standard questions and surveys, and has extensive documentation that can be used for training purposes, has the potential to improve the quality of survey research. That said, abuser-friendly stats software, in my experience, HAS been akin to putting guns in the hands of children in many cases. MR still has a very long way to go in terms of training.

  5. Lots of good comments above. A couple of thoughts…

    Chris, I’ve seen plenty of “experts” in the industry write bad questionnaires or insist on practices that are confounding to respondents. The crisis in panel, which I wrote about in this blog (search “Future of Panels”) is indisputable proof that the full-service industry isn’t doing manifestly better!

    Kevin, I agree about training and user-friendly software. There’s plenty of improvement here, though the industry tends to respond to this with the “better left to experts” refrain, which — in these days of data permeating every corner of business — is not realistic. I think the gauntlet needs to be thrown down in front of the tool providers. This is our point of view at AYTM though. We want our tools to promote good research practices.

    Michael, I’m sure there are DIY success stories elsewhere. We’re looking to create them, too.

  6. You get what you pay for. I can buy a hammer, a saw and some nails fairly cheaply, but I can’t build a house. Quality of sample, questionnaire flow, analysis of results, these all have costs. It’s easy to ask a question. Is it the right question? Are you asking the right people? Are the results statistically significant? Again I’ll use the DIY example, do you do your own plumbing, electrical wiring, or roofing? Or, for those who remember their polling history, Dewey defeats Truman, 1948.

  7. Hi James, not sure I understand the housing example, but while the customers are asking for real time (or at least very fast) responses to their questions. The quality aspects can be as well managed as they can in a “proprietary” MR stack, but without competence you get nothing. I believe MR competence will still be key, but the business models of MR today is up for a shakedown, not because of DYI tools versus traditional models, but because the paying end clients have different requirements. Just look at the investments in technology for CX, Marketing Automation. The industry are not up to the requirements, DYI is just a technology support for a solution, but not alone.

  8. JD – great thought piece!
    I totally agree that the new era of platforms will carve out a nice chunk of the MR market. We’ve already seen this in the Customer Experience Space with MaritzCX and Medalia. Qualtric and other platforms are being common place in corporate MR departments.

    The next generation of researchers are already adopting these survey tools in their firms – I saw this clearly while I was doing a short-term assignment in the Atlanta market. It’s not just about cost reduction – it’s about speed and practicality. Combine the DIY aspect with different levels of service offerings when clients need more horsepower, and a lot of the tactical work will migrate.

    Best wishes on your new role – they are lucky to have you on board.

  9. The “quailty” concerns from these DIY/Instant services are often overstated, in my view, by “traditional” market research organisations who feel threatened by the speed/price of the new entrants in their market.

    Ultimately, it doesn’t matter where the research comes from, there is always the potential for it to be misused, or used incorrectly – it’s equally likely that an Insight Manager will misinterpret the results produced by a full service agency when applying these in a practical sense, as it is for someone to fail to take into account the looser nature of the controls that are in place on a DIY/Instant system.

    The true challenge for these traditional agencies now is to figure out how to combine the two, to provide a more comprehensive package to their clients – immediately springing to mind is using the DIY tool to conduct an “initial” phase, and then taking the findings from this and using it to make a more effective main research project than might otherwise be possible.

  10. Compelling read…and commentary. What our own research we conducted with corporate researchers suggests both sides have things to consider:

    -Agency/MR: clients need answers faster, and researchers who continually reference the past and the need to do things “their way” frustrates clients, as they’re not hearing their business challenge. Corporate researchers aren’t making this stuff up…they’re getting pressure from above and they’re seeking solutions from the companies they know. When those agencies say “no”, “can’t be done,” etc., they start looking elsewhere, as the problem still exists.

    -DIY tools: “this tool can do some neat things…now what?” That is, the tool companies lack expertise. But as pointed out…they’re addressing that by adding talent that can speak intelligently to the client’s design needs. These likely aren’t people without experience…they’re researchers who see a better opportunity partnered with these tools.

    Change is going to continue to happen more rapidly, and new players are going to fill the voids that exist: we’re just now starting to see what many industries are seeing that become more technologically “involved.” Where the MR industry needs to get is realizing if we can’t adequately defend our positions, or change our methods to more ably address the speed of corporate needs (our clients), then we are not going to survive. I regularly see a strong sense of confidence (arrogance?) from both quant and qual researchers that their way is right and the value they bring can’t be replicated…I think we all in the MR industry need to take a step back and reconsider that position.

Leave a Reply to Kevin Gray Cancel reply