Download the

The Quiet rEvolution In Marketing Insights



It’s been a while since I actually posted a blog that I have written. Not for lack of desire or even content to share, but simply because I’ve been incredibly busy with work related to Gen2 Advisors, Insight Innovation Exchange, and GRIT. Thank goodness that there has been a flood of fantastic submissions from our network of GreenBook Blog authors to fill the gaps!

What’s been interesting through these last few weeks of busyness is that despite the varied nature of the projects I’ve been working on, there has been a common theme running through out them all: shifting research models. To be clear, this work hasn’t been driven by preparing for some future change, but rather because the shift has been underway for some time and is now gaining momentum.

This has been fueled by many conversations with client-side organizations, technology companies, research suppliers, and thought leaders that I respect such as Gregg Archibald, Todd Powers, Alec Maki and many others. This post incorporates elements of the various dialogues I’ve been having with these folks in an attempt to synthesize it all.

When looking at this quiet rEvolution in the research/insights industries, start by considering the example of social media. The recent rise of, and fascination with, all things social in the digital ecosystem reflects many of the changes that are sweeping the marketplace.  This naturally touches many other areas of the insight industry, including topics such as big data, mobile, geo-location, online communities and yes, even the humble survey. The transformational impact of the social media age cannot be overestimated; it is indeed pervasive.

To put it in perspective, consider this quote in Digiday from Marc Pritchard, Global Marketing and Brand Building Officer of P&G regarding the company’s view on how technology (social and mobile especially) is changing the game for them:

“To address these [technology] forces, our vision is to build our brands through lifelong, one-to-one relationships in real time with every person in the world,” (emphasis added) Pritchard said. “The power of everyday people is driving monumental change and people power favors brands like ours. We have trusted brands that are part of everyday life. We genuinely care about serving people with superior benefits and doing good.”

“Technology will mean that people will increasingly expect brands to understand their unique needs and deliver,” Pritchard said. “We want P&G to be the first to create this trusted, indispensable relationship because it will create greater loyalty, more purchases across categories, and more sales at lower costs. Achieving this vision requires some fundamental shifts in how we operate.”

The essence here is that social and mobile technologies will be leveraged to drive engagement and understanding of consumers at a previously unimaginable level, increasing basic business value. That has profound implications for the insight organization on both the client and supplier side of the value chain.

The quote above illustrates the role that big data (with social media data being a huge component) will play in helping to inform a significant aspect of the “one-to-one” model by default.  And this will force other changes that are already beginning to play out via new entrants, such as Facebook, Google and Twitter, in the existing MR space.  Consolidation between community providers and consulting organizations, and partnerships between social platforms and research companies are also well underway.

There is a realization in the business community that insights do not arise solely as a product of traditional research models.  Instead, these new platforms are providing tools for discovery and ideation that are faster, more unexpected, more customer-focused and quite often, more productive than existing approaches.

Enterprise organizations acknowledge that the accuracy with which a company can understand its customers determines that company’s success (or lack thereof). But the problem for them is that there are a lot of customers, they have a lot to say, and they don’t necessarily say it clearly. Indeed, the customers themselves may seldom know what they mean, what they want, or what they would do if they got what they wanted. To complicate matters further, customers also experience a wide variety of feelings and attitudes toward the products and services they procure, and often are (1) somewhat unaware of the origin or complexity of their feelings, and (2) do not know how to articulate their feelings in a way that is informative to companies that care about allaying these feelings and improving the experience of the customer.

At the end of the day, companies want to provide for needs and wants in a way that makes economic sense.

The perfect way to do that is to understand a consumer’s individual needs and meet them.  This means having a one-to-one understanding of each of these people, and execute accordingly.  This approach covers the first part of the premise I laid out, but it doesn’t do much for the “economic sense”.

In order to balance the two components, we have said “let’s group consumers by similarities in how they connect with our product”.  Segmentation improves “economic sense” by addressing a larger audience, thus economies of scale.  At the same time it improves “economic sense”, it detracts from providing for needs and wants because the group is never the same as the individual – similar on some criteria, but not the same on that criteria nor similar in other ways.

To gather all the information we need to understand the individual (in the context of a relationship with a product) would have been a difficult task even if it was just a small group of people.  But we are talking about almost 7 billion people.  Those people are different from one another on a huge number of axes – economics, education, values, health, emotions, family, opportunity, culture, physical nature, life stage, housing, etc.

The nature and sheer volume of today’s raw data is the first step on a path that can move us closer to understanding these 6+ billion people.  But the data is still relatively raw and unstructured.  The next major steps are to bring the data together, analyze it, and interpret it into the needs and wants of individual consumers.  We are not there yet.  Nor is the mass customization that is necessary to make “economic sense” out of individual consumer needs.  The interim suggests smaller, holistic, relevant segments – not a bad step and not a pipedream. But bear in mind, it is only an interim step. The ability to bring it all together at scale is rapidly coming .

To date a variety of “traditional” market research approaches or business intelligence approaches have been the primary means of trying to understand these variables. From surveys to focus groups, neuromonitoring to facial scanning, and from data mining to predictive analytics brands have had many tools available to try to understand and structure the process of insight generation.

The issue for brands is complex, but the goal strikingly clear: to understand its customers, they need to listen to them, and not just what they have to say, but also why they feel they have to say it. And, they need to do it in real time and at web scale. To do that, they need a model that allows for the aggregation & synthesis of data into (to borrow a term from my friend Alec Maki) a common information framework.

This has been a challenge for other methods due to issues of cost, scalability, sampling, and hundreds of other business and procedural issues. We are seeing a future unfold right now where the establishment of common information frameworks will function as the central driver of business insights across the organization, and the traditional role of research professionals will evolve to be the facilitators of getting to the “why” of the business issue rather than the collectors of the “who, what, when, where, & how” of data. 

With such realities in mind, Gartner estimates that by 2014, 30% of analytic applications will use in-memory functions to add scale and computational speed; 30% of analytic applications will use proactive, predictive and forecasting capabilities; and 15% of all behavior intelligence deployments will combine behavior intelligence with collaboration and social software and become decision-making environments. Thus, with the requirements for real time assessment, the massive volume of data, and the exponential growth of client expectations, companies’ need to look beyond the status quo for advanced text analytics and high-performance structured data computing techniques. That is the combination that will drive the future.

Don’t take my word for it though: check out this news yesterday: Facebook to Partner With Acxiom, Epsilon to Match Store Purchases With User Profiles. Or two stories last week from Nielsen: Nielsen Ratings Change: Broadband, Console Viewing And More Measurement Coming For TV and Billboard and Nielsen Add YouTube Video Streaming to Platforms. And finally, this gem from Raytheon: Raytheon’s Riot program mines social network data like a ‘Google for spies’. I could cite many more examples, but I think you see where this is going.

What is emerging is the rapid growth of new entrants that combine the platform scale and efficiency with analytical breadth to process vast amounts of aggregated and synthesized data in real time and deliver real insights to clients. These platforms will become centralized data engines that are integrated with multiple data streams to become one hub where all content can be aggregated and processed. Such advancements allow for the real time processing of sophisticated algorithms on huge volumes of both structured and unstructured data that will take marketing insight organizations well into the next decade.

The biggest gap in the industry right now is around this concept of the common informational frameworks. It is the missing contextual piece for big data and is the unifying model that gives MR (in all its forms) a chance to really deliver value in the future. From my work with client organizations the demand is loud and clear: to holistically understand consumers across all brand touch points and to do that they need a real structure to synthesize all of the data coming in from all of the various old, new, and future streams: that is the informational framework. Research suppliers, social media analytics, CRM, etc… just become plugins at that point which deliver discrete value as well as aggregated value.

That is what the MR industry need to be working on, and I think it is a HUGE glaring hole in our collective discourse right now.  The focus is on theory, methods or tech, but not so much in the merger of the three to deliver business value. Many non-MR firms and at least a handful of traditional players are working on this problem to  provide not just the data feeds from many sources, but also the framework to analyze it and deliver context focused on client issues. When these firms get it scaled and the kinks worked out the MR world will change dramatically and many folks will be left scratching their heads wondering where their business went.

The curve has suddenly gotten steeper.  People are changing faster.  Products are being created, or are evolving, at an astonishing rate.  Marketing and market research must simply keep up.  Or be replaced with something else that can tolerate the rate of change.

Please share...

8 responses to “The Quiet rEvolution In Marketing Insights

  1. Lenny, what a thoughtful and well-written piece – thoroughly misguided but beautifully put together. Your argument seems to rest on the back of big companies with big ideas. I’m sure P&G would like to engage with every single person on the planet. That doesn’t mean that people want to engage with P&G, or any other CPG company, for that matter, at anywhere near a meaningful level. I suspect that much of that engagement is of the “give me a coupon” type. You will also find it hard to uncover data that draws the engagement-sales connection. It can’t hurt, but there is little evidence that it matters.

    There are lots of big companies with access to lots of data, making them “Big Data” companies. But it’s just data. IBM is already showing an amazing lack of understanding in its consumer applications (leaving stocking allowances out of their profit per sku calculation, for example). And we as an industry have not seen a good reason to believe any of these other big companies will do any better. That’s on the operator side – on the other side is the that much of the data has not yet been imbued with meaning. Everyone believes, or wants to believe, and I know you’ve drunk the Kool-Aid here, that social media matters. But gee, nobody’s come up with a satisfactory demonstration that likes or mentions or the slickness of a Facebook page matters when it comes to buying stuff.

    We, as an industry, have wanted a holistic understanding of our consumers for at least my 33 years in the business, and I’m sure it goes back farther than that. So the fact that our clients are demanding it doesn’t mean we’re going to give it to them or that they are going to find it. For that matter, it’s unclear that anyone would know what to do with it if we had it.

    But again, eloquently written and kudos for getting to “maybe it’s evolution and not revolution” with the small “r” – I noticed.

  2. Nice to see you back in the blog-saddle, Lenny. I think you’ve made a number of good points here, namely calling attention to the changing nature of the insights business, and reminding us all that as other companies find ways to innovate with these new data sources, it will become more and more difficult to compete with traditional research methodologies.
    My question is: do we need more of this reminding? Those that are more active in the industry should already be aware and working on finding ways to work with these new data sources. I can never quite put my finger on what you propose MR companies do, but your language (not here specifically, just in general) suggests you predict all research techniques like surveys, in-depth interviews, etc. will disappear.
    I think it is far easier to find an example of someone using their data warehouse to address issues that were previously done less accurately through research than it is to take a random client research question and find a data source that holds the answer.
    Either way, as an analyst, I’m excited about data if it can answer the marketing questions of interest. But I’m just wondering when we will stop tearing down the industry and start building it back up again.
    One other question for you (or anyone): should we really be calling analysis of big data ‘market research.’ I think it would be simply data mining and/or analysis. The ‘research’ part implies a specific control over the data collection. Do you agree? (I did notice that the blog title says “Insights”, but still seems like “research” is used interchangeably in our industry).

    1. LOL, Ahh @Steve, I do appreciate your feedback my friend! First thanks for the kudos, and for keeping me intellectually honest (a tough job for sure!).

      I agree that the vote is out on whether consumers will “engage” with brands and what might drive that engagement, but I think that is beside the point. The technology exists and is being brought to market to enable the development of that 1 to 1 relationship, and that process will do much to change the value proposition of MR suppliers, or at the very least where we fit within the new data-driven insights ecosystem. Once the tech is in place, then brands will be forced to deal with consumers in context for good or ill, and that is where I see our profession going; the keepers of the context.

      It all may be bust 10 years from now, but the damage will be done in the interim and MR will be impacted one way or the other. I disagree that the fact that clients are asking for a holistic model doesn’t mean they will find it; basic market dynamics will play out and folks will (and are right now) try to meet the the need. From there marketing science will adapt and we’ll see if real value is derived. My sense is that it will produce value; there are just too darn many smart folks pursuing this goal for it not to work.

      @Dale, thanks to you as well. You’ve really emerged as a great participant in this discourse and it’s always good to get your take. Your question on do we need more reminding is a fair one, and my answer is “yes”. My goal in wiring pieces like this or selecting similar posts for inclusion here on the blog isn’t to beat anyone down, but rather to help our industry understand what is happening and hopefully help point the way for hope to navigate the changes happening successfully. What I want MR suppliers to do is begin the process of adapting to these new realities, or at the very least do some contingency planning, so that they can find a viable path to growth. By exploring the possibilities of what the future holds, I hope that I am helping to make that happen.

      To be clear, I think our profession has a bright future; the need (and ability) to deliver data driven insights has never been greater. I’m not quite so optimistic about our industry as currently constituted or defined, mostly because the business models of most suppliers are based upon data collection as a primary revenue driver, and that model is not sustainable. Much as I may want to, that fact cant be sugar coated. Whether we call the process that we evolve into marketing research or consumer insights or analytical foresight remains to be seen; maybe it will be a bit of all of those depending on the context of the business issue and the data used to solve it. It’s largely irrelevant though because it will all fit under the business intelligence umbrella and that will be a tech driven sector, with humans largely removed from the data collection and analysis process; our job will be to figure out what the analysis means regardless of data source.

      Thanks again guys; good stuff!

  3. Lenny – can’t resist one more shot at you. IMHO, Marketers, like Mr. Pritchard, should be worried about how to engage with people – they should stay up late at night, imbibing on Mt. Adams to get those creative juices flowing. Researchers should be evaluating what Marketers come up with, both before and after the fact. Because many of us are neither siloed nor stupid, we may have some ideas for Marketers to consider, or at least some interesting facts, factoids, unexplained associations, or hunches that can play into future strategies; it’s the stupid Marketer who doesn’t pay attention to someone who knows his/her business well and is bringing information to bear on how to do it better. There’s my ideal world – hand-in-hand marching towards higher market share and profits.
    Your argument seems to be that because technology and data exist, it must prevail. And because there are some big names in the game, it really must and will prevail. And I’m thinking that in the history of marketing research, and I say this as an alumnus, Nielsen hasn’t innovated diddley-squat and is unlikely to innovate anything interesting. Just one example of a name you are bandying about. And there are no outside players in our business who have ever made a substantive contribution. So I’m pretty skeptical about the potential. Remember – we haven’t seen what the likes of Google can actually do because it’s too soon to see the results of research from Google Surveys being implemented. For that matter, we haven’t seen much in the way of Big Data or Social Media doing much of anything for packaged goods.
    So there.

  4. I think it’s a “both AND” situation. Major change is happening AND traditional data collection and analysis is not going away. There is a whirlwind of change happening now as to how marketing is investing in information technology to get at “needle in haystack” information which even a few years ago would have been impossible. At the same time, surveys, focus groups, ethnographies, CLTs, phone interviews and paper ballots are not going away. For example, SurveyMonkey didn’t get an $800M valuation because surveys are dying. At the same time, I doubt SurveyMonkey execs are putting all eggs in the survey basket.

    The question is how will changing marketing information technology (and resulting business implications) AND traditional approaches coexist? What adaptations (rEvolutions), if any, will individuals and companies employ?

  5. Dear Lenny, I normally don’t comment on blogs but took a double take on your “understand a consumer’s individual needs and meet them” and “common information framework” under your banner of the quiet revolution. It sounded like you were writing ‘copy’ for a new-to-market BI choice engine (it’s called Theseus Decision Maker) developed in Australia which it seems might be at the forefront of your quiet revolution. I’m due back in the US (West Coast) next week and wondered if you might have sufficient interest to have a chat about Theseus and choice engines. Basic details at Thank you. Rob

  6. Dear Leonard, thanks for your inspiring paper (and thanks to Steve, either, for comments). The real thing, I would suggest, is what you say at the end: “people are changing faster. Products are being created, or are evolving, at an astonishing rate” . That means simply that the discovery of things like they are is no longer enough. We should predict the future. Difficult task, but not impossible since the future is precisely the basis for every management decision, even for marketing managers. An holistic perspective will help. Researchers would be able to integrate big data, with qualitative research with economic analysis and the like.
    Big data are currently used in the financial industry: with big profits for the happy few, but some disasters once in a while, as we do know. When it comes to predict the future, don’t rely too much in big data and sophisticated alghorithm.

Join the conversation