Our new GreenBook Directory site is live!
Your #1 strategic guide to consumer insights.
Qualtrics X4 The Experience Mgmt Summit
Brand & Retailer tickets for all IIeX events now start at just $99! Get or give one today!

Big Data or Big Hype?

While big data is still in its infancy, early adopters are starting to make progress, particularly in the social media space where they are experimenting with taking conversations and creating meaningful metrics to help brand managers target their messaging.


By Diane Liebenson

Insight Innovation (a GreenBook initiative) recently hosted a webinar, “Big Data or Big Hype.”  Moderated by Lenny Murphy, panelists included:

–           Greg Pharo, AT&T

–           Michael Wolfe, Bottom Line Analytics

–           David Johnson, Decooda

–           David Weinberger Why5 Research

Here are my key takeaways:

  • Big Data offers unprecedented opportunity as well as challenges
  • Using big data to solve business issues is in its infancy, but corporate executives feel tremendous pressure to explore the uses of big data to win real business advantage.
  • There are plenty of examples of Big Data initiatives thanks to early adopters
  • Market Researchers appear to be watching from the sidelines
  • Bridging the gap between the traditional marketing research business model and the Big Data paradigm requires a new set of skills

Defining Big Data

There’s a lot of confusion about the term “Big Data.”  Big Data essentially refers to the fact that a lot more data exists today than even a few years ago and there’s a desire to somehow aggregate all of this data from multiple sources and synthesize it in a way that drives more informed business decisions.

The “4 Vs” of big data illustrate why we are truly in a big data revolution:

  • The Volume of information.  The sheer amount of data is massive.
  • The Variety of data.  Social media is only one piece of big data, albeit a big piece.  Big Data also includes behavioral data, transactional information, videos, photos, GPS locator data, and even voice.
  • The Velocity of data.  The speed at which data can be accessed and analyzed is exploding, enabling analysis of streaming data.
  • The Veracity of data.  None of the above matters if the data can’t be successfully integrated together and synthesized, in real-time, in a way that brand marketers can use to make intelligent decisions.  Matching data on what people say they will do (attitudinal data often captured in surveys) with what they actually do (behavioral or transactional data increasingly captured real-time through mobile commerce) is a game changer.

The Promise and the Challenge

The promise of big data is to tell a story that nobody could tell before, and to tell that story faster than ever.  In essence, big data is about shifting from “What happened? or ‘What is currently happening’ to ‘What do the data mean’ and “What do you predict is likely to happen?’

The analysis of big data is currently in its infancy.  There’s an explosion of data but insights haven’t kept pace with that explosion.

One challenge is to figure out what a conversation actually means, and then translate that into a metric that can be used to analyze customer experience and customer satisfaction.  David Weinberg called this the “holy grail.”

An even bigger challenge is to figure out how to bring together and synthesize multiple data streams.  Organizations have legacy systems – from historical tracking studies, syndicated sources, customer service data, online data from social media, data from mobile, etc. Pulling that data together across different data silos is a challenge whose solution is only just now being addressed.

The Future from a Technology Standpoint

David Johnson from Decooda said that he rarely talks to brand marketers about big data per se.  Instead, he and his team discuss how they can help brands solve business issues faster and better by bringing together data and data sources into one centralized location and in a form that allows all stakeholders to access that data. Providing a 360-degree view of the customer experience, in real-time, is far more useful and insightful than relying on a single data source or using months-old data.

David also noted that most technology firms are math-based and look for patterns to analyze conversations.  A more nuanced view of brand perceptions than a basic ‘thumbs up’ or thumbs down’ is critical and increasingly possible.  By tapping experts with skills in cognitive science, computational math, and linguistics, it is possible to uncover valance beyond sentiment, i.e. emotions like disappointment, irritation, confusion, anger, or outrage.  Understanding what people are most passionately talking about at a very granular level is notably more useful to a brand manager.

Moreover, finding a way to integrate this data with behavioral, transactional, and historical data would create the opportunity to do segmentation analyses that can be predictive of what will happen in the future.

The Business Imperative

While there is some discussion about using big data to build an Artificial Intelligence process, panelists indicated this is not a current reality and may not be for quite some time.

Corporate executives feel big data offers tremendous opportunity and, as a result, early adopters are experimenting with how to use it to best advantage. ROI is clearly an issue as brand manager’s focus on definable business issues such as using big data to improve a marketing or communication program, to assist in crisis management, or to solve some other specific business issue.

Measuring ROI has been difficult because there are multiple touch points in marketing – a buyer can listen to an ad on the radio, go online, visit a store, and then buy.  Michael Wolfe is leading the industry by creating the first ROI metric for social media marketing specifically, and for other types of activities as well.

Examples of Big Data Initiatives and The Companies That are Leading the Pack

Panelists identified several examples of how big data are being used:

  • Researchers can translate social media conversations into basic metrics such as the number of likes, the number of visitors, Klout scores (measures of influence), and descriptive information regarding who is on a social media site.
  • A client might ask: “How do I acquire more customers?”  Big data tools can take a client’s customer satisfaction data, including conversations via social media and mobile, and link them to survey attitudinal data.  A data set with new metrics can be developed from these data, offering a new, more holistic understanding of how to acquire more customers.
  • At AT&T, strategy has traditionally been a top-down exercise using aggregated data.  Increasingly, however, there is value in looking at granular, transactional or account level data. Big data has created the opportunity to take these kinds of data and leverage that information in ways that traditionally weren’t possible for strategic insights.
  • AT&T has used big data to link together data that previously couldn’t be combined.  For example, attitudinal and satisfaction data for questions like “How do you like your carrier?” and “Are we doing a good job on dropped calls?” have been linked to actual experiential data on how well the network actually performed in terms of dropped calls.  Big data’s ability to link these data resulted in a more accurate, holistic view.
  • Coca Cola’s very successful Super Bowl campaign this year integrated data across multiple channels.  The campaign is discussed in an article in Ad Age.

Panelists noted that while some larger companies are doing interesting work, there’s tremendous fragmentation and nobody has a compete strategy. In fact, David Johnson noted that his firm ends up in a co-creation mode with many of their clients because they need help figuring out how new data analytic tools could help them.  As David said, “this is a journey that has only just begun and it’s very much in its infancy.”

Notwithstanding the above, panelists noted that Walmart, Starbucks, and Fedex are doing interesting work and that Coca Cola and Starbucks have a great attitude toward analytics.

The travel industry, particularly the hotel chains and online travel sites, were also singled out for using big data to predict who their best customers are and what types of messaging they should have.  They are using micro-segmentation to target specific messaging to different segments.

Difference between Traditional MR Business Model and the Big Data Paradigm

Greg noted that AT&T typically reaches out to specialists who have experience with big data. As a result, they rarely use their traditional market research clients for this kind of work.

Market researchers run the risk of being marginalized given that the use of market research’s ‘bread and butter’ research techniques – surveys and focus groups – are likely to decline in the face of big data’s ability to capture and synthesize real-time conversations and combine it with data on actual behavior.

One significant challenge that market researchers face is that the skill set needed to deal with the big data paradigm and predictive analytics is broader than that of the traditional market researcher.  Perhaps this is one reason that marketing researchers have been slow to embrace this area.

Four distinct professions were identified as needing to come together to create the data scientist of the future:

  • Traditional analytics
  • Data governance
  • Traditional consumer-facing research
  • Consultancy/business optimization

Those individuals who possess the broad skills of the Data Scientist often have an extensive background in math, physics, and psychology, and they additionally have the ability to derive meaning from data and tell a story.  Because it is extremely difficult to find people with all of these skill, AT&T has created teams that combine individuals with deep data governance experience with those who have consumer research experience.

Greg Pharo from AT&T mentioned that while his department works closely with the IT department to think through issues related to data management/architecture, he emphasized the importance of  having a data governance team within the market research organization itself.


The research tools available as a result of big data are changing, making predictive analytics possible.  This has created a significant and pressing need for Data Scientists, who can unlock the meaning behind big data and communicate that information effectively for business gain. While market researchers are not the ‘go to’ source for insights using this new data paradigm, they nonetheless potentially have the skills to play a role in articulating actionable insights that can be derived from the hose of big data.

While big data is still in its infancy, early adopters are starting to make progress, particularly in the social media space where they are experimenting with taking conversations and creating meaningful metrics to help brand managers target their messaging. Synthesizing enterprise data and bringing it all together remains a challenge that will require significant effort.

Please share...

16 responses to “Big Data or Big Hype?

  1. Diane, a nice summary of the webinar which only highlights how lame most of those who talk about the use of big data really are these days. Why lame? Look at the six examples they cited (after many a promise that exciting applications would finally be revealed). None of these examples show any change in consumer behavior as a function of an analysis of big data (the Walmart story is not true, so that doesn’t count – even your reference says it’s not true). The industry keeps on believing that lots of big companies are doing lots of big things with big data. Nobody can point to an example where a company started making more money by the application of a big data analysis. Until that time, we can all rest easy.

    1. I think the Obama campaign would beg to differ @Steve; they very much used big data to change consumer behavior for a very tangible result. Target is also an example with their much talked about pregnant teenager story. I think the reality here is that either companies are not discussing their successes out of competitive concerns or we’re not plugged in to the right outlets to hear them. Follow the money; investors and clients are spending huge amounts in this category (far more than MR global spend); I doubt that is happening with no tangible results.

  2. @David – so Sprint harvested some social media data and built a model around this to predict churn, figured out where they could change their processes to reduce churn based on the model, and it worked out well. For anyone who has been doing text analytics for any length of time, this is what you’ve been doing – not sure why this is Big Data, unless it’s because it used lots of data rather than a little data. And if the amount of data is our criterion for a classification as big data, can we all go back to sleep now – that would be a pretty meaningless distinction.

    @Lenny – I give you the Obama campaign as an example of resource allocation improved via big data. Target just took their own sales data and did some analysis based on a request from marketing. Not to diminish the accomplishment, but again, they could have done this years ago if they had the computing power. Without the computing power, they could have narrowed down the scale to customers in a smaller geographic area, reducing the sample to a manageable level and specifying the model, then testing the model on hold-out samples to increase confidence.

    Again, a definition of big data that is based on “there’s lots of it” is a pretty lame definition. And to assume that something will be big and useful because lots of big companies are throwing lots of money at it is just silly – there’s no logic to that (see single source data, see Project Apollo, etc.).

    1. But computing power is part of the equation; sure the theory is nothing new, but the ability to make it real is. Now we’re in the experimental phase of learning how to harness the data to produce results. It’s a brave new world my friend…

  3. @Steve – You are correct we helped Sprint use hundered of millions of consumer conversations combined with internal data to uncover an insight that drove millions of dollars of incremental revenue. I am surprised to here you say that that is not big data. What is your definition? If you are interested I will be sharing my definition of big data at the Kellogg School KIN conference next week. Cheers David

  4. IMHO, Big Data is interesting or threatening, depending on where you sit, when it is combining different data sets tapping into different aspects of a subject/buyer/shopper/user, in a way that wasn’t able to be done before, because the information wasn’t available or combine-able. I am avoiding the “bigger is better” or “more is better” fallacy in any definition. You could have done the work with hundreds of conversers or thousands of conversers – you chose to do it with millions. Nothing wrong with that – but it does not make it any more valid or interesting from a research perspective. What’s interesting and exciting is you came up with a usable model that had dramatic impact, not that you used millions of conversations.

  5. So the caipirinhas have worked their way out of your system, Lenny, and you’re seeing clearly now? No, computing power is only a factor in the brave new world of MR when it lets you do something you couldn’t do before. In the text analytics area, this could have been done years ago with the computing power everyone had. You would just have to sample and maybe replicate against hold-out samples – not a big deal and not a game changer.
    And remember, my young friend, that Brave New World was a dystopia. You would do well to keep a little of that 1984 paranoia handy 🙂

  6. I think a huge part of the problem is that the concept of “big data” is so new that no one really knows what it is or how to use it – it’s a bunch of people putting their toes in the water and still defining things. I’m sure we don’t hear about all the success stories, but believe me, we don’t hear about all the failures either.

    So AT&T married actual customer database data to survey data – good for them. I did that at Bank One in 1994 on our brand tracking study and it changed the way we did the research. Was I just 19 years ahead of the times, or is that simply not what “big data” actually is?

  7. @Ron – thanks for that point about what you did. All the stories (except for Obama, Lenny – I’m conceding that) I hear are people doing stuff I was doing in the 80s and 90s at Burke, IRI, and Nielsen. We may have been a little ahead of our time, but not 20-30 years so.

  8. I think that brand and marketing leaders are always looking to improve the speed and application of market research data. It seems to me that the term “Big Data” in and of itself has an identity and image problem. There appears to be much confusion and mis-perception around what it is and what it is not. Moreover, the term just sounds intimidating….as in “oh no; more mounds of data to sift through!” To me it’s kind of like the buzzword du jour and the 21st Century variation of “Big Blue”. In this sense and in reading the posts here, I tend to align more with Steve Needel’s perspective and agree that bigger data is not necessarily better data.

  9. Steve, I believe I cited on example where a client redesigned a promotion from social media content analysis we did and the spectacular results achieved. I think you have to listen for these things.

  10. Just to share my view of where “Big Data” fits.

    Currently I see Market Research as a 10 year old GPS system. It will take you from point A to point B as long as you put in the correct information and drive sensibly.

    The addition of “Big Data” sources brings that GPS system up to date, it still takes you from point A to point B if driven sensibly, but it has the added benefits of other sources of information to help you get to your destination more easily, much as current GPS systems understand traffic congestion and road works.

    Big Data to my mind doesn’t replace market research but enhances and compliments it at the same time..

Leave a Reply to Steve Needel Cancel reply