Research Methodologies

July 26, 2012

When Marketing Research is Like a Sunset On Pluto

Imagining a sunset on Pluto is difficult due to our incomplete knowledge. Why do we ask respondents questions we know they can’t answer?

Joel Rubinson

by Joel Rubinson

President at Rubinson Partners Inc

0

 

By Joel Rubinson

Gregory Berns, distinguished neuro-economist says, “the brain is fundamentally a lazy piece of meat”.

He demonstrates the principle by asking us to imagine a sunset on a beach and then to imagine a sunset on Pluto.  He asks us to note how much harder the latter is to do.  Also, while you get answers, they are highly inaccurate.  So, when is marketing research like asking people to imagine a sunset on Pluto?

Recently, I have seen a report on people’s prediction of their comfort with the idea of stores becoming just showrooms a number of years from now where people actually make their purchase on their smart phone. Really? Consumers can predict the future like that? That is just one of many Pluto studies I see regularly reported on in the trade press.

Behavioral economics studies the shortcuts we take to simplify our decision making when faced alternatives for which we have varying degrees of incomplete knowledge and gives us clues as to how surveys can take a really wrong turn.  Think about how someone fills out a survey when some of the questions don’t quite work.  They are making choices about how to answer questions and the less natural the question or harder to recall the right answer, the more they are using short cuts that are predictably irrational, as Dan Ariely might say.  For example, when people are asked which brands of cereal, they bought in the past three months and we compare answers to actual tracked behavior, we see significant overstatement of brand penetration for every brand.  If I first ask about the past year, past six months, and then the past three months, I will get much closer to the truth.  What cognitive and emotional processes are leading people to access their memory in a faulty way resulting in what is called “telescoping”? How about private label brand preferences? They are always understated.  People buy private label brands much more than they admit in a survey.  Why and what can you do about it?

Think about brand attribute ratings batteries.  Every tracker and A&U has them.  What happens when someone is responding to a phrase that does NOT describe how they personally think about the brand or even choices in the category?  They guess at the answer!  They go through a brief self-analysis, translating the way they DO think into an answer for the question as asked.  The respondent level test-retest reliability of such answers is really low (similar to why 50% of respondents often do not get classified into the same attitudinal segment if the questionnaire is re-administered).

Here are seven tips for avoiding marketing research about Pluto

  1. Don’t ask questions about behavior if you can measure behavior directly. This is especially true today when digital and shopping behavior is so easily tracked.
  2. Ask questions with natural vocabulary, revealed by social media listening.  How do people naturally talk about your brand, category, and need states? Change your questionnaire and attributes to reflect emerging vocabulary and don’t worry so much about preserving trends on irrelevant phrases.
  3. Only ask people to recall ephemeral behaviors (like shopping, or using a smart phone) right at that moment
  4. Make the survey answer choices mirror the real life purchase or consumption choice you are trying to research as much as possible.  That is why constant sum approaches do a better job at predicting an individual’s probability of buying a brand than 5 point scale purchase intent questions. In real life, shoppers choose among brands, not purchase intent answers for a given brand (as Eric Marder has noted in his book The Laws of Choice).
  5. Nudge (as in the behavioral economics book) respondents’ answers about private label or anything else that is repeatedly underreported.
  6. Keep surveys short.  A series of hard questions will have a cumulative effect at depleting a respondent’s energy and make their “guesses about themselves” even worse as the survey goes on.
  7. Use lead-ins and warm-ups so respondents know what you are after. While this has been frowned on by many purists as potentially biasing, it actually anchors the respondent by giving them a framing, points of comparison, and categorization to work from.  It gets their head and heart into the moment that the research refers to, which according to George Lowenstein another noted behavioral economist, is critical. People want to be helpful; equip them to give you the right answers!

Marketing research needs to be mindful that behavioral economics principles apply to the dynamics of survey taking.

0

behavioral economicsneurosciencesurvey design

Disclaimer

The views, opinions, data, and methodologies expressed above are those of the contributor(s) and do not necessarily reflect or represent the official policies, positions, or beliefs of Greenbook.

Comments

Comments are moderated to ensure respect towards the author and to prevent spam or self-promotion. Your comment may be edited, rejected, or approved based on these criteria. By commenting, you accept these terms and take responsibility for your contributions.

More from Joel Rubinson

Are You Using Synthetic Data for Analytics?

LevelUP Your Research

Are You Using Synthetic Data for Analytics?

Explore the use of synthetic data to bridge the gap between sales and ad exposure data. Learn how it can enhance targeting and validate ad effectivene...

Joel Rubinson

Joel Rubinson

President at Rubinson Partners Inc

How Baseball Led Me to Marketing Analytics

LevelUP Your Research

How Baseball Led Me to Marketing Analytics

Discover the power of Moneyballing marketing and revolutionize your outcomes by leveraging math over judgment for superior results.

Joel Rubinson

Joel Rubinson

President at Rubinson Partners Inc

Brand Differentiation: A Reality the Dirichlet Model Can’t Account For

LevelUP Your Research

Brand Differentiation: A Reality the Dirichlet Model Can’t Account For

Reevaluate marketing strategies and research approaches by challenging the Ehrenberg Bass Institute's claim on brand differentiation.

Joel Rubinson

Joel Rubinson

President at Rubinson Partners Inc

How Can You Tell If Your Advertising is Working?

LevelUP Your Research

How Can You Tell If Your Advertising is Working?

Learn how to increase measurement sensitivity with five principles of ad effectiveness measurement. Don't let flawed approaches hinder advertising suc...

Joel Rubinson

Joel Rubinson

President at Rubinson Partners Inc

ARTICLES

Moving Away from a Narcissistic Market Research Model

Research Methodologies

Moving Away from a Narcissistic Market Research Model

Why are we still measuring brand loyalty? It isn’t something that naturally comes up with consumers, who rarely think about brand first, if at all. Ma...

Devora Rogers

Devora Rogers

Chief Strategy Officer at Alter Agents

The Stepping Stones of Innovation: Navigating Failure and Empathy with Carol Fitzgerald
Natalie Pusch

Natalie Pusch

Senior Content Producer at Greenbook

Sign Up for
Updates

Get what matters, straight to your inbox.
Curated by top Insight Market experts.

67k+ subscribers

Weekly Newsletter

Greenbook Podcast

Webinars

Event Updates

I agree to receive emails with insights-related content from Greenbook. I understand that I can manage my email preferences or unsubscribe at any time and that Greenbook protects my privacy under the General Data Protection Regulation.*