Human Recall is Not Perfect: A Quantified View of Self-Reported vs. Behavioral Data

Editor’s Note: This post is part of our Big Ideas series, a column highlighting the innovative thinking and thought leadership at IIeX events around the world. Bonnie will be moderating a client panel at IIeX North America 2019 in Austin, TX. If you liked this article, you’ll LOVE IIeX North America. Click here to learn more. 

---

Marketers have long relied on consumers’ memories to measure and predict shopper behavior through online surveys. Yet, it is no secret that human recall is far from perfect. Think about your own shopping experiences. Can you remember all of the products you looked at or bought when you bought them, and what price you paid?

DISQO commissioned an independent market research firm to put shopper recall to the test. Our key objective was to help determine when we can best apply a standalone survey-driven methodology, and when it’s preferable to leverage observed behavioral data instead. When does recall really start to break down?

The Study

To achieve this, we asked consumers shopping-related questions via survey, while also tracking their digital activity using proprietary technology. We quantified the “recall gap”, defined as the difference between what consumers recalled and their corresponding behavioral data. This enabled us to begin to measure where the largest gaps exist and consider why.

The research methodology entailed surveying approximately 700 consumers; they were surveyed in December 2018. Research participants were asked about their past 30-day shopping for Electronics and Personal Care products on Amazon and Walmart.com.

Panelists responded as they would to any traditional online survey. The consumers in our study were validated shoppers in specific categories of interest, eliminating the potential for inaccurate survey-based screening.

The Results

The findings validate that, in most cases, there is a divergence between what consumers recalled doing online and their corresponding behavioral data. However, the accuracy of consumers’ recall varies by the type of shopping behavior being measured. While recall of general online behaviors is better, recall significantly breaks down with specific behaviors.

Let’s start with the broad categories. Consumers accurately recalled shopping for Electronics on Amazon or Walmart.com in the past 30 days— Respondents thought they shopped less than their behavioral data demonstrates by 4 percentage points–certainly nothing to be concerned about.

The difference between consumers’ recall and actual behavior for the Personal Care category is 12%. Yes, that’s a larger recall gap than Electronics, but relatively small compared to more specific categories.

For example, within Electronics subcategories, consumers thought they shopped significantly more than their behavior suggests. Respondents over reported their shopping behavior for all three subcategories included in the research, TV/Media Players, Cell Phones/Accessories and Computers/Accessories, by at least 22 percentage points. The same pattern is seen for the three subcategories measured for Personal Care (Hair Care, Skin Care, and Oral Care).

Consumers also had trouble accurately remembering if they shopped on Amazon or Walmart for Electronics and Personal Care products in the last 30 days. However, Amazon fared better than Walmart in this assessment for both categories in terms of respondents’ recall.

A recall gap exists when looking at actual purchase activity and comparing what price consumers paid for products using behavioral vs. survey data. The difference between the price recalled and the actual price was as high as 15 percentage points.

The metrics evaluated in the research-on-research demonstrate the complexity of shopper research and the potential flaws of relying solely on self-reported survey data for shopper behavior.

The Next Steps

Where does that leave us? What steps can be taken by marketers and researchers who rely on self-reported shopper data?

Fortunately, today there are new solutions available to provide a bridge between human recall and actual behavior. Marketers and researchers can now utilize first party, permission-based sources of factual, observed digital behaviors as the basis for shopper research.

Why “screen” consumers by asking if and when they purchased a computer on Amazon, or any website when we already know the answer? Not only does this eliminate the guesswork, it positively impacts consumer engagement. The use of behavioral data reduces the time consumers spend answering screening questions that only add to what is typically a long, laborious survey experience.

And let’s not stop there. Why not tap into the upper and lower funnel to understand consumers’ digital behaviors before and after they made a purchase? In addition to accurate data, behavioral data provides the opportunity to discover the full consumer journey which will lead to tactical, highly actionable insights.

Access to digital behavior is exciting and should be leveraged. The combination of observed behavioral interactions to understand “what” people actually do and the ability to survey consumers throughout their journey to understand “why” provides a new and powerful toolkit, ultimately leading to better business decisions.