Night Thoughts of Classical Market Researchers

The core challenges for market research’s future aren’t as much about the technical skills of practitioners as they are about their mindsets. Other fields are bringing new kinds of more relevant thinking to marketers struggling with a digitally transformed world.

It has been nearly three years since I retired from the world of Market Research.  While I am partial to retirement, I do like to keep up with the field through consulting, reading, talking to old friends and attending the occasional market research conference.  A few weeks ago I attended a very different conference than the “usual” market research conference – BRITE (Brands, Innovation and Technology) at Columbia University. The juxtaposition of this with some of the more typical market research conferences I’ve attended recently brought to mind an unusual novel I read over 30 years ago – “Night Thoughts of a Classical Physicist”, by Russell McCormmach.

McCormmach, a historian of science, wanted to write about the difficulties older, established German physicists in the early 20th century had in accepting the “new Physics” of Relativity and Quantum theory.   The difficulty wasn’t so much technical (although the math was different) as it was mental; their physics was mechanical, a world of Newtonian absolute motion through the ether.  These German physicists just couldn’t process that their worldview and, all they had worked on in their careers, was fundamentally flawed and being swept away. Instead of writing a typical history, McCormmach felt that the best way to convey these mental struggles was by writing a “novelized” history.  His protagonist, Victor Jakob, a professor at a small university, is a fictional character, but is actually a compendium of several real physicists in the German university system of the time. All through the novel, Jakob’s “night thoughts” and interactions are based on actual events and writings, and are carefully documented in the endnotes.  While not a novel that will make anyone forget Fitzgerald, the novelistic device produces a book that is moving portrait of an older scientist struggling with a new world.

As I reread the novel I couldn’t help but think about the current state of market research and where it is headed.  The usual market research conferences, while generally good, are for the Victor Jakobs. BRITE, on the other hand, not a market research conference, is really more for the Albert Einsteins.

Let me explain.  While the market research conferences I have attended over the last year or two have featured good solid, professional work, the work is still too grounded in traditional market research thinking  – a mindset of individual survey studies, not programs that integrate multiple sources of data. Of generating data, not leveraging data clients already have to generate dollar value. And, of thinking that their job is to explain, not to predict.    Most presentations I’ve seen haven’t quickened my pulse or left me feeling that the industry has cracked the code on helping companies prosper in this transformed digital world. Except for the inclusion of new survey technology or buzzwords, most, frankly, could have been given 5 or 10 years earlier.  

Happily, I have seen a few presentations that tried to push the envelope by being based on, or at least integrating non-survey data.

For example, at IIeX last year, Joel Rubinson did a really interesting presentation showing that incorporating panel data in targeting algorithms can dramatically increase the effectiveness of digital advertising campaigns.  And Sarah Snudden (Keurig Green Mountain) along with Kirsten Zapiec and Melinda Smith (bbb Mavens) showed how integrating disparate, unstructured data streams leads to a deeper understanding of consumers emotions and behaviors.  

More recently at Quirk’s Brooklyn, Renee Smith of Gutcheck (a former Kantar colleague) and Kristen Bull of Heartland Foods showed how the integration of survey data and big data allowed them to gain insights they would not have gained through only one type of data.  What struck me, though, was that Renee seemed to almost plead with the audience to think of doing something similar in their own practices. It brought to mind an earlier session at the conference that was an audience “debate” over whether big data was going to replace market research by 2038 – not a lot of real informed debating, but a lot of deer-in-the-headlight looks in the audience.

I felt very differently when I attended BRITE a few weeks ago.  As I said before, BRITE isn’t a market research conference, which is why I always find it refreshing.  Speakers tend to be CMOs, entrepreneurs, academics and technologists – people on the cutting edge of their field.  While sitting in the audience at the end of the first day, I had a sudden realization – I was watching the replacement for a lot of traditional market research, even if they weren’t calling it that.

One speaker was the CEO of Foursquare, who explained that their business is very different than what it was a few years ago.  Their location technology is now embedded in a lot of other applications (want to know where that little moving dot on that Maps app on your smartphone comes from?).  They are actively working with clients to help them generate value from the location data they are producing – sounds to me like they could call themselves a market research company if they wanted to.  There was an expert panel on blockchain, and unlike other such discussions, the focus wasn’t on crypto currencies, but rather about how a variety of other industries, including marketing, are starting to take advantage of the technology.  A third session was by the COO of a consulting company called Teradata that builds systems for companies that allow them to leverage the multitude of data they generate, analyze it at scale, and drive monetary benefits. And there were others of a similar vein.  

Will “traditional” market researchers be able to play in the world I heard about at BRITE?  Some will. When I was still at Kantar, there was a small group of brilliant people scattered throughout the world working together (sometimes informally) with vision and drive to bring this kind of thinking to their clients.  I’m sure there are pockets in other companies as well. But they are only pockets.

There are strong technical headwinds facing traditional market research companies – they just don’t make enough money to invest in R&D they way they need to, while VCs are pouring billions into AI, predictive analytics, et al.  These headwinds could theoretically be overcome with the right kinds of partnerships and acquisitions, but it won’t be easy.  Individual market researchers should be investing in themselves through certificate programs in data science. But all the money in the world can’t change the “night thoughts” of most market researchers.  Which is why there will be fewer of them over time. There are others, who wouldn’t call themselves “market researchers” and with very different mindsets and skillsets, stepping up to the plate.

Please share...

3 responses to “Night Thoughts of Classical Market Researchers

  1. a few random comments…
    1. i too attended the quirk’s event in brooklyn last month and was part of the big data vs market research debate, namely would the former replace the latter in 2038.
    i was the sole person who stood up and said ‘yes’, at which point the moderator reworded the question and asked about the increasing role that big data would have!
    folks in the room were visibly defensive and somewhat frightened that their jobs would disappear in less than 20 years.

    2. from the time i studied mtmm (multi-trait, multi-method) in college, i have utilized my own version to constantly seek different tools to measure behavior and see where things converged — or not.
    in other words, never rely on just quant or qual or one group of respondents. seek ways to examine a problem from different approaches. yes, more time and money, but often better insights and predictions.

    3. is all the new panel research, big data analytics, biometric measurement, internal and external data (e.g. from social media, location based apps), etc just another name for “market research”? after all, what is truly being studied if not a market? there is no other term under which all of the other ones can be placed.

    4. i agree 100% that attending conferences, webinars, trade shows and the like that cover topics from different fields and industries can be enlightening. it gets one out of their comfort zones and helps one to observe what other functions are interested in and care about.

    5. “market research” does not need rebranding so much as repackaging if the discipline is to attract new, young practitioners.

  2. Thanks Larry for the great insights and for the shoutout. There IS something more that research needs to find. My current mantra is that we need to impact marketing productivity..not with ahas that bear vague relationship to outcomes but with real and measured improvement in marketing ROI short term and long term. This requires mastery of activation space and the data sets that it relies on. On the other hand, many media side data scientists don’t know sh*t from Shinola about consumer dynamics. There is a fusion of thinking that can benefits both worlds.

Join the conversation