The ultimate guide to transforming your business with business insights
Insights That Work

The Modern Research Respondent: Holding Their Attention with Dynamic Questions

While there are many points that must be addressed to significantly increase engagement, Focus Vision recently studied the specific impact of dynamic questions. Here is what they learned.

Man with megaphone and white board


By Aaron Jue

It’s a constant struggle in the world of market research – keeping an increasingly demanding respondent engaged. With attention spans at an all-time low, and technology innovations across the board, people expect and feel entitled to a certain level of interactivity while online. This includes the interface that they encounter when responding to market research outreach.

While there are many points that must be addressed to significantly increase engagement, we recently studied the specific impact of dynamic questions. These questions are more compelling and interactive, providing a graphical and interactive way for researchers to capture respondent data in online surveys. Standard HTML inputs, such as traditional radio buttons or select boxes, may not be the most user friendly or engaging forms for today’s research respondent. Using Javascript or HTML5, dynamic questions enable more flexible and customized design elements to hold respondent attention and address the need for greater responsivity.

Dynamic element examples include:

  • Drag and drop
  • Sliders
  • Card sort
  • Rank sort
  • Button select

The enhanced and flexible design capabilities of dynamic questions allows researchers to create question types that are suitable for different platforms. Traditional radio button designs may be too small for mobile device screens. We can use dynamic question to create larger, mobile friendly forms, such as ATM style boxes.  Visual cues– like the illusion of a depressed button when it is selected–also enhance usability and help increase survey participation among the mobile population.

Instead of a pick list using checkbox forms, the survey researcher may employ a shelf test to better simulate a real-world shopping experience. Using a drag and drop interface, respondents can physically rank order a set of cards rather than typing in a rank number in a text box. In both of these instances the dynamic forms establish an intuitive and user-friendly way for survey respondents to express their opinions about a brand or product.

But dynamic questions executed poorly can increase dropout rates, reduce tendency to read questions, and even cause respondent confusion when not executed properly.  We have seen designs where a respondent was navigating a space ship and had to “shoot” an item to indicate a response; or were timed to move objects through goal posts. In these instances, an additional layer of cognitive processing involved hand-eye coordination and manipulation with the computer mouse. The resulting data was sporadic as respondents focused more attention on the act of completing the task at hand rather than thinking about the question asked by the researcher.

There are common sense ways to avoid pitfalls when using dynamic questions:

  • Pre-testing: This is a critical step for any elements that stray away from standard question structures. Does the question work on multiple platforms (PC, mobile, tablet)? Is it time intensive for the respondent? Is it self-explanatory so the respondent knows what to do?
  • Teamwork: Good design also requires specialized expertise. While many market researchers may direct hired programmers on how they want their dynamic questions customized or setup, this task is better done in collaboration with a visual designer or usability specialist.
  • Standardization: Data collected by using dynamic question may not be directly comparable to data collected using traditional formats. Any comparisons with previous norms should be done with careful consideration. When employing dynamic questions, develop standards and maintain consistency in how you customize them. That way you can properly interpret the relative value of your data without worrying that the question format is biasing results.
  • Clear Benefit: Your dynamic question may require extra clicks, additional mouse movement, or dragging and physically manipulating objects across the screen. Is there a clear and substantial benefit to having respondents spend this extra time? In other words is your dynamic question that much better than a traditional HTML form?

Dynamic questions can improve the survey experience. As researchers build surveys on multiple platforms and screen sizes, dynamic questions allow the designer greater flexibility to meet this challenge. Dynamic questions offer additional functionality for online surveys where graphical and interactive displays allow the research to capture information in more engaging ways that are not possible in a traditional format. But understand, too, that if not done properly, building new question types can create confusion as respondents encounter something they are unfamiliar with and may use them incorrectly or not in the intended manner.

Please share...

4 responses to “The Modern Research Respondent: Holding Their Attention with Dynamic Questions

  1. Great tips – not just for creating dynamic questions, but for any survey design process. Is there any data anywhere showing what best practices are for using dynamic questions, whether they actually lead to increased response rates, whether they provide better data than traditional question types (especially drag and drop versus typing in a rank order number)?

  2. The importance of the author’s point about “clear benefit” can’t be stressed enough. With regard to some recent developments in online survey GUIs, I’ve found that animation for the sake of respondent engagement may or may not translate into benefits for the respondent and/or the client. One example I’ve seen recently is a radio button that slides into place when you click it, vs. just changing color to indicate that the response has been selected. It achieves its purpose of engaging the respondent, but almost completely eliminates the focus on the question itself. This is of particular concern in surveys where the content is more dense and the questions are more complex. Also, some dynamic effects that work well on the survey builder’s larger monitor may not be nearly as effective when using responsive design and viewing the same effect on a smartphone or tablet.
    When survey vendors task their development teams with designing new types of question displays, I hope that the starting point is “what can we do to create question formats that will improve the collection, validity and reliability of respondent data” vs. “what can we do to engage respondents”. That way, respondent engagement becomes one component of the product development goal, vs. the primary one. Then, test the new dynamic question type(s) against the same question in non-dynamic form, echoing Z Johnson’s comment regarding supporting data and best practices.

  3. Personally, I’ve found click-to-rank much easier for respondents than drag-and-drop ranking – particularly for touchscreen devices where the touch, hold, drag, release sequence needs to be done quite carefully.

    Done well, dynamic questions get to answers that wouldn’t be possible using traditional methods so don’t just look for comparability. For instance our hot-cold aka thumbs-up/thumbs-down questions give a natural and intuitive way of combining choice and scale-type data that aren’t possible in any other format.

Join the conversation


Aaron Jue

Development Director, Electronic Frontier Foundation