Five Winds of Change, Sweeping Away Traditional Market Research

Last week, I spent five wonderful days at the ESOMAR Congress in Edinburgh, UK. The week started with an all-day Council meeting on Saturday, workshops, and pre-meetings on Sunday, and two-half days of presentations, meetings, and exhibitions on Monday, Tuesday, and Wednesday.

There were too many interesting things to provide a comprehensive report, but they can be summarised in five key themes:

1. Automation and AI is all around us
2. Redefining our Domain
3. The Great Ethical Debate
4. ‘No Questions Research’
5. The Skills we need.

 

Automation and AI is All Around Us

At Congress, there was a raft of presentations that included the use of automation, data blending, and artificial intelligence. The tools of these papers covered topics such as chatbots, image processing, data curation, voice, and predictive modeling. However, most of these papers were not about automation, data blending and/or AI, they were about business results, new products, new marketing, new customers, societal change, etc. The Automation, AI, and data blending were being reported as a means to achieve the topic being discussed – in most cases, they were not the topic itself.

The big picture is that these tools are being used widely and they are delivering results. They are not being used widely in the sense of most companies using them, but widely in the sense that the absolute number of companies using them is large. The exhibition showed a similar picture. Most of the services and technologies on offer were using some combination of automation, AI and data blending, but most of them talked about the results, not the process.

Key takeaway: If you are not heavily using these topics in your work, you are probably being left behind.

 

Redefining Market Research

The term market research has served our industry well for nearly 100 years, but changes in the use of data, technology, and business practices are causing a widespread discussion about the definition of ‘our business’. Many of the companies we are competing with do not define themselves as being in ‘market research’. When a CEO says “We are going to leverage our data lakes to become customer-centric”, they do not automatically think ‘I should be leveraging market research’. However, this is exactly where our understanding of people, how to research people, and how to turn information about people into insights, should be taking us.

Key Takeaway: Our domain (i.e. the business we are in) is the use of data from and about humans (ranging from big data to ethnographic observations) to make better business decisions. If we think of a business as being the intersection of supply-side management (eg, building factories, employing workers, designing products etc) and demand-side management (eg, marketing, pricing, sales etc), then our domain relates to demand-side management. We turn human-centric data into insights that drive outcomes.

My feeling is the term ‘market research’ should be allowed to be replaced with something that conveys using data about people to make better business outcomes. The name may be something new, or it may be as simple as ‘Customer Insights’. Yes, I know that some of our work does not relate to customers (eg, voters, citizens, patients, employees etc), however, the bulk of the money will relate to customers and remember that when we called our industry market research, it was not all about markets.

 

The Great Ethical Debate

The issues surrounding fake news, Cambridge Analytica, and mis-selling have brought the topic of ethics to the forefront of many discussions, by the media, citizens, and leaders of business. This discussion has spawned a number of legislative initiatives, such as GDPR in Europe and the California Consumer Privacy Act in the USA.

Market research has a long history of addressing ethics and there are a large number of codes and guidelines that have been created by the research trade bodies (such as ESOMAR, MRS, ASMRS, AMSO etc). One consequence of market research’s codes and the wider context of scandals and laws is that a growing number of data-intensive industries are attracted to the codes that have been generated by market research. There is an opportunity for us to set the pattern for the ethical use of data.

However, there seems to be an awareness that, in addition to codes of conduct and methods of adapting to new laws, there is a need for a deeper review of ethics. In a connected world, where nothing is able to remain hidden or anonymous, what are the core principles that should apply? Questions include: What does informed consent mean? To what extent can people revoke their consent, and how should they be able to do it? How should the rights of the individual and society be balanced? To whom do we owe a duty of care?

Key Takeaway: Over the next few years, expect to see moves to create a more informed debate, more codes, and more laws. Also, expect to see more scandals. This trend interacts with the move to expand the boundaries of what we consider to be ‘our business’. As we find more data scientists, data controllers, and marketeers within our domain, the more our traditional rules will need to be reassessed to see if they are fit for purpose. For example, one change that is being widely discussed is a move away from anonymity towards informed consent.

 

‘No Questions Research’

From my analysis of the ESOMAR Global Market Research report, it is clear that traditional ‘Questions Research’ (surveys, focus groups, and depth interviews) is becoming a smaller and smaller part of the market research and insights pie. It seems likely that ‘No Questions Research’ is already about 50% of the market research and insights ecosystem (in terms of value).

‘No Questions Research’ includes passive data, secondary data analysis, analytics, social media listening, CCTV systems, transactional data, etc. Note: not all ‘No Questions Research’ is quantitative – for example, traditional ethnography is often ‘No Questions’ and most semiotics is both qualitative and ‘No Questions’. My take, from the data-centric conferences I have attended over the last year, is that there is a growing appetite for what has been called ‘survey enhanced models’, where surveys (and I would suggest qualitative research) are used to improve models created from ‘no questions’ data. Similarly, there is a growing opportunity to blend ‘no questions’ data with traditional market research to improve the accuracy of the research and to reduce the number of questions we need to ask.

Key Takeaway: ‘Questions Research’ appears to have been largely static in size over the last few years, which means that, once factors like inflation and currency changes are taken into account, it has slightly reduced in size. It is likely that ‘Questions Research’ will remain broadly at its current size (about US $40 billion) for the foreseeable future. By contrast, ‘No Questions Research’ is also about $40 billion USD and is growing at about 5% a year. This growth will appear much larger if the boundaries of insights are broadened to encompass more of the new uses (most of which are ‘No Questions Research’).

 

The Skills We Need

The skills that market research and insights people need are changing. We have been aware of the need to increase our consulting, storytelling, and business skills for some time. There has been a growing awareness that what we need are T-shaped people (this is true of many other industries and professions too). The T-shaped employee has a broad base of skills (for example they understand the core principles of qual and quant research) and they have a deep specialty (the stick of the T).

Key Takeaway: In the context of the T-shaped employee, there is an almost endless list of things in which people can specialize, including storytelling, ethnography, semiotics, data science, project design, programming, data visualization, etc.

However, there are skills that are likely to be in short supply, including data science, programming, data visualization and the ability to implement and manage AI systems. The two biggest challenges are that these people are expensive and that they can readily move from market research/insights to a wide variety of other industries and sectors. We need to make market research/insights a more attractive place to work if we want to attract and retain these people.

 

In Summary

The traditional core market research skills (surveys, focus groups, and depth interviews) are becoming a smaller and smaller part of our industry. The name ‘market research’ is probably a limiting factor – we need to focus on the sorts of problems we should be solving, using data about people to help organizations make better decisions. Changes in our role, along with changes in society, mean that we need to update our ethical debate to ensure we remain relevant and effective.

Because our focus is based around understanding people, our skills need to be focused on doing what machines can’t do and combining that with the things machines can do, to provide better outcomes.

 

Editor’s Note: One of the highlights of the Fall conference schedule is the ESOMAR global congress. Here, Ray Poynter – recently appointed Chief Research Officer with Potentiate – gives us his summary of the major takeaways from the conference. None of these should be major surprises to those who’ve paid attention to developments over the past ten years, but it is satisfying to see how mainstream they’ve become – to the point, as Ray argues, that the very term “market research” may be obsolete. We’d love to hear comments from others on their feelings about this potential change.
This article was originally published on Potentiate.