The ultimate guide to transforming your business with business insights
Insights That Work

Paradigm Shift: From recall to occurrence

Tim Rotolo discusses the challenges of recall-based feedback and the benefits of remote user testing.

Lights of ideas

Image Credit: Saad Faruque

The human capacity for recall is subject to a number of major psychological flaws. The only way to close the gap between what testers do on your site and what they tell you they did on your site is to listen to their thoughts in real-time.

Recall is, quite simply, the retrieval of information stored in the memory. People rely on recall all the time, reaching into the brain’s vast catalogs to answer questions like: Where did I leave my phone charger? Which gas station has the best prices? What filename did I use to save my document under?

In researching our customers, we frequently rely on recall-based feedback from regular users, recruited testers, and focus group members, through surveys, direct questioning, or other methods. We ask things like: What did you think of the registration process? Did you have any trouble finding the contact information? How did the landing page make you feel about our brand?

The problem is that methods like this force users back to their mental caches for the answers, and this exercise is fraught with psychological flaws. A number of factors exert influence on the accuracy of human recall:

  1. Attention – If the respondent’s attention was divided at the time they encoded that memory, their recall later on will be hampered. So for example, if your survey respondent was using your website in one tab, and shopping for Mother’s Day gifts in another, and listening to music, their recollection of the experience is almost sure to be a little mixed-up.
  2. Motivation – When no incentive exists for the respondent, their feedback is likely to be less accurate. Alternatively, when an incentive does exist but emphasizes completion over accuracy, respondents are more likely to provide inaccurate, and ultimately counterproductive, information.
  3. Primacy & Recency Effects – People tend to be worse at remembering the middle elements of a series (or an experience) than they are at remembering the first and last bits. In other words, between the very first impression and the most recent memory, things tend to get hazy. This also makes it easy to overestimate the importance of early- and late-occurring problems in the user flow relative to other issues.
  4. Interference – Recall suffers when time has elapsed between the encoding and the remembering, especially if other actions were taken in the meantime. If respondents aren’t filling out your survey right after they finish using your product, everything they do in between affects their ability to respond accurately.
  5. Context & State Dependency – Recollection is impaired when the respondent is in a different environment or mental state than when they encoded that memory. If they used your product at work, for example, but filled out your questionnaire at home in front of the TV, their answers may not reflect their actual experience.

In addition to all these, respondents may also be affected by issues related to response formats or settings. The framing of the question, for example, can have an enormous influence on people’s answers. If your respondents are participants in a focus group, their feedback may be manipulated by social pressures; people give in to conformity and bandwagoning, they lie to avoid appearing inferior or incompetent, they answer questions the way they believe they are “supposed” to, feeding the administrator what he or she wants to hear.

The human mind does endless twisting and turning, self-deceiving, and overcompensating to satisfy the situation – and as a result, almost none of your recall-based feedback is left untouched by one or another distortions.

There are ways to control for some of these issues; problems like division of attention of context dependency might be diminished by confining testers to a controlled environment, but at the cost of losing a genuine, true-to-life look at the customer’s experience. Primacy, recency, and interference could be combatted by asking testers questions at the end of each individual task, but at the cost of obstructing the natural flow.

Shifting to Occurrence

The only way to close the gap between what people do or feel and what they remember doing and feeling, and what they tell you they did and felt, is to look at their occurrent thoughts – that is, the thoughts that are in their mind at the exact moment of their actions.

By eliminating the need for recall, we eliminate the flawed mental filters, the forgetfulness for middle elements, the transitions between mindsets and environments. Verbalization of the thought process, as it happens, becomes the vehicle for higher-quality feedback.

This is something that comes naturally to people: almost all of us talk to ourselves at times, especially when alone. Now companies are harnessing this natural instinct to create the next generation of customer research. Remote user testing taps into testers’ screens and microphones to capture real-time experiences, allowing researchers to listen to their real-time, occurrent thoughts and get the accurate, unadulterated picture.

Keeping things remote has a number of benefits too:

  • Testers provide feedback in a comfortable, real-life environment – in their own home or office or other frequented space, using their own device that they’re familiar with.
  • Testers are not affected by a compulsion to please the researcher or render the “right” answer since there is no personal connection or interaction between the two.
  • Unhindered by matters of logistics, researchers can access testers anywhere in the world, who are able to provide feedback on their own time.

For all of these reasons, there has been a palpable shift from recall-reliant feedback methods towards occurrence-based methods, and the young remote user testing industry has seen massive growth in the past few years – evidence that on a large scale, companies and researchers are rethinking how to get the best data.

The problems with recall have long been known to psychology, but a change in paradigms can only occur when there is something to switch to. Now, with services that connect researchers to users around the world and help them understand what people do and why they do it, recall-centric thinking can and should be laid to rest. After all, why keep peering through a cognitive kaleidoscope when you could be peeking into your customers’ heads?

Tim Rotolo, UX Researcher, TryMyUI
Tim Rotolo, UX Researcher, TryMyUI

Tim Rotolo is a UX Researcher at TryMyUI, a remote usability testing company based in the California bay area. You can reach him at [email protected] or follow @trymyui on Twitter.

Please share...

4 responses to “Paradigm Shift: From recall to occurrence

  1. You know, Tim, we psychologists have a name for people who talk to themselves. Actually, we have a number of names. Schizophrenic and Bat-Shit Crazy are two terms that come to mind. 🙂

  2. Big assumption here that your respondents get into a stream of consciousness that overcomes rationalizations. Why would that be the case, since this is surely just another experimental situation? If it was that easy we could ask everyone to just jabber freely about their attitudes to brands and get great insights. Maybe usability is one area where you can get close to reduced rationalizations?

  3. Chris –

    You make a valid point, and its true we can never get rid of all user rationalizations. I have seen users forgive, overlook, or make excuses for poor design more than once during test videos. But I do think we get pretty close to a real stream of consciousness by having people test on their own devices in their chosen environment, without a human moderator. Getting rid of that human-to-human interaction makes users feel less obliged to flatter and deflect.

    Anyways, it is also true that even on their own, in non-test environments, users will still make some of the same rationalizations that we see in tests. It does not always, or necessarily, reflect the influence of being in an experimental situation.

    Thanks for your comment!

  4. Actually Tim, now I think about it, it is probably as good as you can get because the test environment is totally familiar (home, office) and once they are engaged they will forget about any moderation factors. I think for usability testing this is a great tool. And the big plus undoubtedly is the real time observable feedback. Great tool!

Join the conversation