Editor’s Note: Today we have the capability to capture behavioral data via experiments as well as observed/monitored at a scale and cost efficiency unimaginable just a few years ago. The concurrent growth of new thinking and learning in the fields of neuroscience, psychology, sociology, data science, etc… certainly call into question the efficacy and best uses cases of tools used in the past to deliver insight into human decision making. In today’s post (an admittedly provocative one!), Anoaur El Haji pushes the debate forward again. Many readers will have strong opinions about the premise here, but it is an important discussion as we continue to explore the evolution of our industry from a methodological and contextual perspective.
Being human means being quite curious about other humans. Specifically, we’re very interested in why people do the things they do. The better we understand each other’s motivations, the better we can serve each other. We know that at the most basic level (1) beliefs and (2) preferences determine to a great extent what people do. These are the informational building blocks that shape human decision-making.
(Un)fortunately, beliefs and preferences are hidden by default. Despite all the technological advances, humanity hasn’t even come close to inventing a device that would allow you to read anyone’s mind. So instead we resort what seems to be the next best thing: simply ask people about what they believe and value, what researchers call survey research.
Survey research only makes sense if and only if people honestly report their beliefs and preferences. The value of survey research is directly linked to this fundamental assumption. It’s a fact of life, however, that we have the ability to misrepresent ourselves. And often there are reasons to do so. For example, your willingness to pay for a new luxury watch will probably depend on who’s asking. You might overstate the amount to impress friends while you would downplay it to negotiate a good deal with the salesman.
Because people are free to misrepresent themselves, it raises the question of whether surveys can provide an accurate view about what people truly believe and value. Sadly, there’s solid evidence that surveys are unreliable and give a skewed picture. The problem is so systematic that there’s a whole body of scientific studies focused on what’s called the hypothetical bias.
The root of this problem is that talk is cheap. In a survey, there are no consequences to misrepresenting yourself. The problem becomes even worse because we like to tell what people want to hear, also known as the Hawthorne effect. The end result is that survey measurements of beliefs and preferences are often significantly biased. Compare this to making a purchase. If you buy something that you don’t want, you’re going to regret making that decision. So there’s a strong incentive to make decisions that correspond to your true beliefs and preferences. Actions speak louder than words.
This doesn’t mean that all survey questions cannot be trusted. There are no reasons to misrepresent, for example, your gender or highest completed level of education. In fact, the answers to these types of questions can be verified objectively. However, questions that require value judgments or reporting beliefs are susceptible to bias because these are inherently subjective.
The science of humans is the only field in which the subject matter is able to talk back. So it’s quite tempting to simply trust what people claim about themselves. This shouldn’t, however, prevent researchers from maintaining a high standard to get a reliable view of the world.