Most Predictions are Flawed, but it’s Important to Keep Making Them

On Tuesday 5 March I will be on a panel with Lenny Murphy, Lucy Davison, and Melanie Courtright assessing the latest trends in market research and insight. This panel session will be broadcast live as part of the Festival of NewMR (it’s free and you can register here).

Predictions and assessments of trends are one of the most popular topics. GreenBook’s list of its most popular articles frequent highlights articles on trends and forecasts. However, most of these assessments are under or over-estimates, they are consistently flawed – but they are also an essential part of navigating the fast-changing world of market research and insights. The researcher and author Philip Tetlock has been researching predictions since the 1980s and has concluded that most predictions fail, that most experts do worse than a chimp with a pin (i.e. worse than chance) and that predictions more than a few months out are particularly flaky. If you are interested in the topic of predicting and forecasting then I strongly recommend his book Superforecasting: The Art and Science of Prediction.

Why Predictions Can Be Useful

I firmly believe that making predictions, and urging other people to make predictions, can be a really useful exercise, despite the poor track record that predictions have. In order to assess the ways predictions can be useful, we need to separate the different categories and motivations for forecasts and predictions. So, here are seven categories/motivations – and the benefits they can bring.

1. The Planner

Plans, essentially, consist of three elements:

1. A prediction of what is going to happen, for example, demand for X will increase, the price of Y will decrease etc.
2. An aspiration of where we want to be, we want to grow the volume and gross margin of our business by focusing on X.
3. A set of actions that we think will help us achieve our plan.

A typical plan will include targets, for example, targets for sales by month, for new offices opened, and for new staff recruited. These targets are a prediction of what might be possible. However, they are not a passive prediction, they are targets that need to be worked at in order to achieve them. They are as much about creating the future as predicting the future.

During the year the business will measure its performance and compare it with its plan. Three things may happen:

1. The organization may be meeting its targets, in which case the plan will stay as it is.
2. The organization may be exceeding its targets. In this case, the organization needs to assess why this is happening and normally will want to modify its plan, to take advantage of the situation.
3. Quite often the targets will not be met. In this case, the organization, if it is wise, will investigate why and develop a Plan B.

In the case of the business plan, the predictions and the planning mean that the organization can work together to achieve shared goals. By comparing outcomes with targets/predictions the organization has a reference point. The alternative to having a business plan is to simply react to changes in the market, a strategy that is rarely successful, and very is often very stressful.

 

2. The Evangelist

The evangelist believes in something and will tend to make forecasts consistent with their beliefs. Examples of these people are the people promoting the Internet of Things, Paleo Diets, Neuromarketing, Bitcoin revolutions, the sharing economy and so forth.

According to Tetlock, these forecasters are amongst the least accurate. The more emotionally attached we are to a topic, the less good we are at forecasting it.

For the rest of us, these forecasts have two useful elements. Although this group is less accurate than a chimp with a pin, they will sometimes be right. Their descriptions of what will happen can create a set of early-warning indicators that help us see that something is a change, not a fad.

The early promoters of the Internet, Social Media, and Smartphones were all evangelists – but they turned out to be right.

The evangelists can also serve a role in making change happen. Part of the predicting process can be a conscious or unconscious effort to make a change happen. When Naomi Klein wrote her book No Logo, it was partly her prediction of a less commercial world, but it was also an attempt to move the world in that direction.

 

3. The Commentariat

The commentariat is a slightly pejorative description of people whose role it is to make comments and forecasts, particularly via the media. This is another group that Tetlock says are particularly poor at making accurate predictions. IMO, the key reason these people are so poor at making correct predictions is that they have to create entertainment value. The two forecasts that are most likely to be true are:

1. No change
2. A change that is currently happening and that everybody else is forecasting.

The professional opinion-vendor cannot afford to offer either of these forecasts very often, which means they are fishing in a pool of less probable outcomes.

The value to the rest of us of these predictions and forecasts is that it helps shape the debate about a topic and it indicates areas of thinking that we should understand. In terms of changes, the number of things we don’t know is close to infinite, so it is useful to have people highlighting lines of inquiry that we might find helpful, for example in terms of increasing our understanding of a topic.

 

4. The Lobbyist

Of all the predictors, Tetlock described these as the least accurate. In some ways, the lobbyist is like the evangelist, but the key difference tends to be that the lobbyist is focused on creating an outcome, not on being accurate. When the spokesperson for a political party says that the unpopularity of their party will soon end and that voters will come back to them, they are trying to influence that outcome, whether or not they believe it. The forecasts issued by a company when it launches on the market with its IPO are intended to create success, not simply predict it.

Indeed for a lobbyist, the likelihood of something happening is a restriction, not a goal. The claims made cannot be so crazy as to make the message ineffective. For example, a company selling programmatic advertising cannot credibly say that it will replace all other approaches, so the claim may be scaled back to something like ‘massive growth’ or ‘all leading brands will use it’.

 

5. The Science Fiction Writer

Science fiction writers do not set out (usually) to accurately predict the future. Although in many cases that is exactly what they have done: e.g. Jules Verne predicted the journey to the moon in 1865, Aldous Huxely’s Brave New World predicted antidepressants and many of today’s ills in 1931, and in 1948 George Orwell’s 1984 predicted the surveillance state.

Science fiction often takes one or more changes and then explores how that would shape society. This process can be a useful way to try to innovate new ideas and to help think about how new ideas could be integrated into modern life. This use of predictions is a combination of helping with ideation and helping know how best to communicate a possible future.

 

6. The Scenarioist

Many futurists do not try to accurately predict a specific future, they are well aware that they are unlikely to be correct. Rather than one specific future, the scenario planner creates alternative futures and then looks to:

1. Create early warning indicators to help determine which scenario seems to be happening.
2. Develop strategies that are applicable to several scenarios – reducing the risk of future shocks.

Scenario planning is much less sexy than predicting a specific future because the scenerioist is rarely exactly right, but they are rarely completely wrong. A good introduction to scenario planning is David Schwartz’s The Art of the Long View.

 

7. The Data Scientist

In recent years there has been a massive growth in the use of predictive analytics. These are statistical processes used to forecast the outcomes of new campaigns, election results, and more complex things such as the weather.

A great introduction to the use of data for making predictions is Nate Silver’s book The Signal and the Noise, where he explains why baseball and the weather are relatively predictable, and why soccer and earthquakes are not.

The key thing to remember about most predictive analytics is that they assume the future is a complex collection of straight-line changes from the present. As Nassim Taleb highlights in his book The Black Swan, this is not always true. My personal prediction is that predictive analytics will be really useful in the future, but not in the next 5 years.

 

Predicting Project Outcomes

Before closing this post I want to share one of my key uses of predictions. My core business is helping organizations conduct research, either to help them understand current markets and customers or help predict what would happen if they took a specific course of action (such as increasing prices, run a new ad campaign, or launch a new product).

At the scoping stage of the project I usually ask everyone to make predictions about what they think the outcome will be, and I make predictions myself. The reason I do this is to understand where people are coming from, what their expectations are, and to ensure that the research I do will either:

1. Confirm their beliefs.
2. Challenge their beliefs.

When the project is complete, comparing the predictions to the outcomes can provide great insight into what we know and what we don’t know. Insight into things that seem to be working and things that need to be challenged and changed.

 

The Case for Predicting

So, I hope you will agree that even if most predictions are inaccurate, there is merit in making them. The process of predicting the future requires us to examine the present, it enables us to think about what we want the future to be and can help us be quicker at interpreting change when it happens.

To quote the American general and President Dwight Eisenhower “I have always found the plans to be useless, but planning is indispensable”.

Join Trending Tuesday for Updated Predictions

For more predictions and assessments, join Lenny Murphy, Lucy Davison, Melanie Courtright and me assessing the latest trends in market research and insight. This panel session will be broadcast live as part of the Festival of NewMR (it’s free and you can register here).