Night Thoughts of Classical Market Researchers

It has been nearly three years since I retired from the world of Market Research.  While I am partial to retirement, I do like to keep up with the field through consulting, reading, talking to old friends and attending the occasional market research conference.  A few weeks ago I attended a very different conference than the “usual” market research conference – BRITE (Brands, Innovation and Technology) at Columbia University. The juxtaposition of this with some of the more typical market research conferences I’ve attended recently brought to mind an unusual novel I read over 30 years ago – “Night Thoughts of a Classical Physicist”, by Russell McCormmach.

McCormmach, a historian of science, wanted to write about the difficulties older, established German physicists in the early 20th century had in accepting the “new Physics” of Relativity and Quantum theory.   The difficulty wasn’t so much technical (although the math was different) as it was mental; their physics was mechanical, a world of Newtonian absolute motion through the ether.  These German physicists just couldn’t process that their worldview and, all they had worked on in their careers, was fundamentally flawed and being swept away. Instead of writing a typical history, McCormmach felt that the best way to convey these mental struggles was by writing a “novelized” history.  His protagonist, Victor Jakob, a professor at a small university, is a fictional character, but is actually a compendium of several real physicists in the German university system of the time. All through the novel, Jakob’s “night thoughts” and interactions are based on actual events and writings, and are carefully documented in the endnotes.  While not a novel that will make anyone forget Fitzgerald, the novelistic device produces a book that is moving portrait of an older scientist struggling with a new world.

As I reread the novel I couldn’t help but think about the current state of market research and where it is headed.  The usual market research conferences, while generally good, are for the Victor Jakobs. BRITE, on the other hand, not a market research conference, is really more for the Albert Einsteins.

Let me explain.  While the market research conferences I have attended over the last year or two have featured good solid, professional work, the work is still too grounded in traditional market research thinking  – a mindset of individual survey studies, not programs that integrate multiple sources of data. Of generating data, not leveraging data clients already have to generate dollar value. And, of thinking that their job is to explain, not to predict.    Most presentations I’ve seen haven’t quickened my pulse or left me feeling that the industry has cracked the code on helping companies prosper in this transformed digital world. Except for the inclusion of new survey technology or buzzwords, most, frankly, could have been given 5 or 10 years earlier.  

Happily, I have seen a few presentations that tried to push the envelope by being based on, or at least integrating non-survey data.

For example, at IIeX last year, Joel Rubinson did a really interesting presentation showing that incorporating panel data in targeting algorithms can dramatically increase the effectiveness of digital advertising campaigns.  And Sarah Snudden (Keurig Green Mountain) along with Kirsten Zapiec and Melinda Smith (bbb Mavens) showed how integrating disparate, unstructured data streams leads to a deeper understanding of consumers emotions and behaviors.  

More recently at Quirk’s Brooklyn, Renee Smith of Gutcheck (a former Kantar colleague) and Kristen Bull of Heartland Foods showed how the integration of survey data and big data allowed them to gain insights they would not have gained through only one type of data.  What struck me, though, was that Renee seemed to almost plead with the audience to think of doing something similar in their own practices. It brought to mind an earlier session at the conference that was an audience “debate” over whether big data was going to replace market research by 2038 – not a lot of real informed debating, but a lot of deer-in-the-headlight looks in the audience.

I felt very differently when I attended BRITE a few weeks ago.  As I said before, BRITE isn’t a market research conference, which is why I always find it refreshing.  Speakers tend to be CMOs, entrepreneurs, academics and technologists – people on the cutting edge of their field.  While sitting in the audience at the end of the first day, I had a sudden realization – I was watching the replacement for a lot of traditional market research, even if they weren’t calling it that.

One speaker was the CEO of Foursquare, who explained that their business is very different than what it was a few years ago.  Their location technology is now embedded in a lot of other applications (want to know where that little moving dot on that Maps app on your smartphone comes from?).  They are actively working with clients to help them generate value from the location data they are producing – sounds to me like they could call themselves a market research company if they wanted to.  There was an expert panel on blockchain, and unlike other such discussions, the focus wasn’t on crypto currencies, but rather about how a variety of other industries, including marketing, are starting to take advantage of the technology.  A third session was by the COO of a consulting company called Teradata that builds systems for companies that allow them to leverage the multitude of data they generate, analyze it at scale, and drive monetary benefits. And there were others of a similar vein.  

Will “traditional” market researchers be able to play in the world I heard about at BRITE?  Some will. When I was still at Kantar, there was a small group of brilliant people scattered throughout the world working together (sometimes informally) with vision and drive to bring this kind of thinking to their clients.  I’m sure there are pockets in other companies as well. But they are only pockets.

There are strong technical headwinds facing traditional market research companies – they just don’t make enough money to invest in R&D they way they need to, while VCs are pouring billions into AI, predictive analytics, et al.  These headwinds could theoretically be overcome with the right kinds of partnerships and acquisitions, but it won’t be easy.  Individual market researchers should be investing in themselves through certificate programs in data science. But all the money in the world can’t change the “night thoughts” of most market researchers.  Which is why there will be fewer of them over time. There are others, who wouldn’t call themselves “market researchers” and with very different mindsets and skillsets, stepping up to the plate.