The ultimate guide to transforming your business with business insights
Insights That Work

Why Has Social Media Analytics Met With Limited Success In Market Research?

The weakness of social media analytics is that it can't answer most of the questions that market researchers' clients ask.


Editor’s Note: It’s well established that Ray Poynter and I see the world of research in fundamentally similar ways, but we also disagree often on specifics, especially around future projections. Generally I think our differences are driven by the nature of our jobs and geography: I work with clients and start-ups in an advisory role almost daily and Ray is engaged with MR practitioners and is focused on leading the development of best practices. In the case of this post though, our differences are not wide.

In today’s post Ray does a great job looking at the high level successes and challenges to date in social media analytics fitting in the MR wheelhouse and has a reasonable view on where the market is going. However, I think there are more examples of deeper strategic insights success than accounted for here, and a much broader category of businesses playing in this market.

Early last year Gen2 Advisors (my consultancy) released a report on this sector and it has a wealth of information that will add to this topic. As a special perk for GreenBook Blog readers, you can download it for free here: Gen2 Report From Online Chatter To Meaningful Insights


By Ray Poynter

OK, let’s get one thing clear from the outset; I am not saying social media mining and monitoring (the collection and automated analysis of quantitative amounts of naturally occurring text from social media) has met with no success. But, I am saying that in market research the success has been limited. In this post I will highlight a couple of examples of success, but I will then illustrate why, IMHO, it has not had the scale of success in market research that many people had predicted, and finally share a few thoughts on where the quantitative use of social media mining and monitoring might go next.

Some successes
There have been some successes and a couple of examples are:

Assessing campaign or message break through. Measuring social media can be a great way to see if anybody is talking about a campaign or not, and of checking whether they are talking about the salient elements. However, because of some of the measurement challenges (more on these below) the measurement often ends up producing a three level result, a) very few mentions, b) plenty of mentions, c) masses of mentions. In terms of content the measures tend to be X mentions on target, or Y% of the relevant mentions were on target – which in most cases are informative, but do not produce a set of measures that have any absolute utility and usually can be tightly aligned with ROI.

An example of this use came with the launch of the iPhone 4 in 2010. Listening to SM made it clear that people had detected that the phone did not work well for some people when held in their left hand, that Apple’s message (which came across as) ‘you should be right handed’ was not going down well, and that something needed to be done. The listening could not put a figure on how many users were unhappy, nor even if users were less or more angry than non-users, but it did make it clear that something had to be done.

Identifying language, ideas, topics. By adding humans to the interpretation, many organisations have been able to identify new product ideas (the Nivea story of how it used social media listening to help create Nivea Invisible for Black and White is a great example). Other researchers, such as Annie Pettit, have shown how they have combined social media research with conventional research, to help answer problems.

Outside of market research. Other users of social media listening, such as PR and reaction marketers appear to have had great results with social media, including social media listening. One of the key reasons for that is that their focus/mission is different. PR, marketing, and sales do not need to map or understand the space, they need to find opportunities. They do not need to find all the opportunities, they do not even need to find the best opportunities, they just need to find a good supply of good opportunities. This is why the use of social media appears to be growing outside of market research, but also why its use appears to be in relative decline inside market research.

The limitations of social media monitoring and listening
The strength of social media monitoring and listening is that it can answer questions you had not asked, perhaps had not even thought of. Its weakness is that it can’t answer most of the questions that market researchers’ clients ask.

The key problems are:

  • Most people do not comment in social media, most of the comments in social media are not about our clients’ brands and services, and the comments do not typically cover the whole range of experiences (they tend to focus on the good and the bad). This leaves great holes in the information gathered.
  • It is very hard to attribute the comments to specific groups, for example to countries, regions, to users versus non-users – not to mention little things like age and gender.
  • The dynamic nature of social media means that it is very hard to compare two campaigns or activities, for example this year versus last year. The number of people using social media is changing, how they are using it is changing, and the phenomenal growth in the use of social media by marketers, PR, sales, etc is changing the balance of conversations. Without consistency, the accuracy of social media measurements is limited.
  • Most automated sentiment analysis is considered by insight clients and market researchers to either be poor or useless. This means good social media usage requires people, which tends to make it more expensive and slower, often prohibitively expensive and often too slow.
  • Social media deals with the world as it is, brands can’t use it to test ads, to test new products and services, or almost any future plan.

The future?
Social media monitoring and listening is not going to go away. Every brand should be listening to what its customers and in many cases the wider public are saying about its brands, services, and overall image. This is in addition to any conventional market research it needs to do; this aspect of social media is not a replacement for anything, it is a necessary extra.

Social media has spawned a range of new research techniques that are changing MR, such as insight communities, smartphone ethnography, social media bots, and netnography. One area of current growth is the creation of 360 degree views by linking panel and/or community members to their transactional data, passive data (e.g. from their PC and mobile device), and social media data. Combined with the ability of communities and panels to ask questions (qual and quant) this may create something much more useful that just observational data.

I expect more innovations in the future. In particular I expect to see more conversations in social media initiated by market researchers, probably utilising bots. For example, programming a bot to look out for people using words that indicate they have just bought a new smartphone and asking them to describe how they bought it, what else they considered etc – either in SM or via asking them to continue the chat privately. There are a growing number of rumours that some of the major clients are about to adopt a hybrid approach, combining nano-surveys, social media listening, integrated data, and predictive analytics, and this could be really interesting, especial in the area of tracking (e.g. brand, advertising, and customer satisfaction/experience).

I also expect two BIG technical changes that will really set the cat amongst the pigeons. I expect somebody to do a Google and introduce a really powerful, free or almost free alternative to the social media mining and monitoring platforms, and I expect one or more companies to come up with sentiment analysis solutions that are really useful. I think a really useful platform will include the ability to analyse images and videos, to follow links (many interesting tweets and shares are about the content of the link), to build a PeekYou type of database of people (to help attribute the comments), and will have much better text analytics approach.

Please share...

28 responses to “Why Has Social Media Analytics Met With Limited Success In Market Research?

  1. Thanks, Ray…interesting and relevant post

    In my opinion, social media monitoring is the literal example of “Big Data in the wild.” As such, it entails all the challenges facing Big Data adoption—quality/cleanliness of data, certain data siloed/inaccessible, requiring significant sr. management support, etc. The focus on value of the data often trails that of volume and velocity.

    From a market research perspective, the challenge is taking it “out of the wild” into a more controlled environment in which the researcher can understand where the un-structured data are coming from, who is saying what, and in what context.

    To address these fundamental challenges will require, on an ongoing basis, small-scale test-and-learn experiments to understand context, as well as quality and predictive accuracy of the coded un-structured data in answering specific research questions. There are ways to feasibly do this by creatively combining certain research methods with big data applications to refine the text analytics tools currently available. A caveat: taking this small scale test and learn approach to define and extract value from the data is probably perceived as less “sexy” compared to cracking the social media mining code. Therefore, I’m not sure if many companies are taking this approach—the absence of which may be a reason contributing to the lack of success you alluded to.

  2. Crimson Hexagon focuses on the analysis on unstructured text, such as social media. Instead of automated sentiment, CH uses a patent human-trained statistical algorithm developed at Harvard that gives researchers the flexibility to beak down a conversation in any way, taking into consideration the nuances and context of a conversation. There are still many challenges in slicing the data into demographic segments, as many of this information is not available about online audiences.

  3. Well I am sure Lenny expected me to come in on this one, because I seem to have been one of the lone voices with a seriously jaundiced view of the potential in social media over the last 3 years, a period when everybody was jumping on the bandwagon talking about revolution and the demise of market research as we know it.

    I approach the problem with social media from two perspectives. Firstly we need to understand that social media is ridden with biases and this is what concerns me immensely and should be worrying to buyers of research. Analysis of Twitter communications by a number of academics has highlighted the intentional bias from the paid bloggersphere. Also the participation rates are heavily biased to under 30 year olds and females. Even with the massive improvement in text analysis tools the output seems to be unexciting. As Ray pointed out, great for confirming message delivery, assuming the paid bloggers haven’t been ramping the numbers.

    Secondly and slightly related, Marketing Directors are highly suspicious of the analytics, as well they should be. It is proving increasingly difficult to find any correlates between recall and sentiment and anything that these marketers understand, like buying behaviour. Even major brands like P&G and Coca Cola are wondering where the sizzle is from major online campaigns. Yes there are occasional great stories, but are these outliers or at the end of teh day a function totally of their creativity rather than brand messages? Marketing people I have talked too find the major appeal is the low cost of utilizing these media vehicles. Most have doubts about their efficacy but as a small slice of total media spend hell why not look trendy?

    Renato’s comments suggest the social media research industry is really at very early stages of credibility. If we have to go back to focused testing it suggests the hype over the last few years has been grossly misplaced. As one skeptical finance company marketing director said “its like being at a football game. When everyone stand up, you also have to stand up”. Methinks the low cost, availability and mass reach tail is wagging the logical dog here and will for some time.

  4. I totally agree with Ray Poynter’s position. The strength of social media monitoring and listening is that it can answer questions you had not asked, And the main limitation is in that they don’t allow for mapping and undrstanding the whole space. Sound it familiar? Of course it does. We found here the same qualities we were used to assign to qualitative research. Remember? Focus group, in depth interviews and the like. That appears like a great oxymoron: qualitative research was associated with small numbers. But this isn’t true, and never has been in the past. Social media analytics should be developed and accepted in the same way. It is clear we should give up any claim for “reliability” . At their best they could be considered “valid”, but not “reliable”.
    Thank you Ray for your remind. And thank to Renato Silvestre either for intriguing suggestions
    Carlo Erminero

  5. Ray has given us an excellent checklist for social media listening impact. And he is right that most listening approaches fall short.. But not all; some have effectively addressed these points.
    1. Point: Most of the comments in social media are not about our clients’ brands and services…
    Counterpoint. The value of social media is that it gives a perspective of a brand in the context of people’s lives, rather than in the context of a product category.
    2. Point: It is very hard to attribute the comments to specific groups, for example to countries, regions, to users versus non-users – not to mention little things like age and gender.
    Counterpoint: Agreed this is critical.Converseon (to whom I advise) has technology to segment social media profiles based on forensic signs over time as to their geography, gender, age, brand preferences, and lifestyle interests. In fact, we can even match social media conversations to customer segments that a client established from surveys or customer data.
    3. Point: The use of social media by marketers, PR, sales, etc is changing the balance of conversations.
    counterpoint: This is a strength and one of the few places where marketing research can directly translate insights into action. We listen, we engage via social marketing, we observe effect.
    4. Point: Most automated sentiment analysis is considered by insight clients and market researchers to either be poor or useless.
    Counterpoint: It’s true that most off the shelf SaaS systems have crappy accuracy (maybe 60%) which is definitely useless. Some use much more sophisticated language processing that can get accuracy in the 80-90% range vs. human coding. My guess is that this is equal or better vs.survey results if you were to interview the same person twice one month apart or administer the same survey to different online panels.
    5. Point: Social media deals with the world as it is, brands can’t use it to test ads, to test new products and services, or almost any future plan.
    Counterpoint: social media senses emerging signals. It is much more foresight oriented than brand trackers where one is reluctant to change attribute lists. Can’t test ads? you are only limited by imaginative design.

    As to Ray’s future vision, well mine is different. I don’t see a future where we only advance the qualitative agenda. I believe social media has strong quantitative properties that can be demonstrated via regression and resampling. How marketers can be social is a pretty important discussion, one where research can get a seat at the table and even lead the meeting because social media insights come not from a sample regarding this issue, but from something closer to a census.

  6. one more point:
    one this, Ray and I agree: “There are a growing number of rumours that some of the major clients are about to adopt a hybrid approach, combining nano-surveys, social media listening, integrated data, and predictive analytics, and this could be really interesting, especial in the area of tracking (e.g. brand, advertising, and customer satisfaction/experience).”

  7. There is a clear (but not always appreciated) difference between “conversation monitoring” — which is what most basic social listening platforms do in a pretty simplistic manner and “conversation mining” which applies new technologies and approaches to get to meaningful insights. The latter is an areas of rapid advancement with new text analytics solutions that scale human intelligence through machine learning to take sentiment (and emotion, purchase stage, intent) analysis to the next level reliability and to better understand “who” is having the conversations. Trying to retrofit a basic monitoring to do deep insights is where researchers go off the tracks — it’s a good starting point but there are other technologies, like ours, that “start where listening platforms stop” that already address many of the issues above. It’s a fast changing world, and those interested in this space should keep a close look on where it’s headed because most of those objections are already being addressed. More to come.

  8. Ray I believe the reasons why social media listening and analytics has not taken off among researchers and insights professionals yet are:

    – most social media monitoring tools – with a few exceptions – have been developed with the PR Manager in mind not thinking of the consumer insights manager
    – as a result their sentiment accuracy has been typically below 60%. Nowadays accuracy of 85% and above is being achieved with automated tools by a few companies focussed on solving this problem
    – it is difficult to eliminate noise in an automated way – but it is not impossible
    – the existing tools were not good at all in multilingual high sentiment accuracy
    – the consumer insight managers were sceptical because of all of the above. This will change as more and more credible tools will enter the market.

    We all need to find ways to better integrate traditional research with listening, some ways you already describe in your article. Once the accuracy and the integration barriers disappear the listening disciplne will take off as a fully accepted market research and insights tool.

  9. Ray has made some excellent points. I’d like to add a few comments if I may.

    ‘Social Listening’ as such is still a very immature technology/industry when compared with more established media and forms of monitoring. Despite its hoopla and eye watering valuations, there is still a lot of improvements that need to be made for this to become more embedded in MR process.

    As with any young industry the first to market can become the biggest which is different to being the best. As market competition intensifies I would hope to see more innovation than simple derivation of the ‘social listening’ platform.

    We should also consider that ‘social media analytics’ as defined in the title is much more than social listening. If you take Twitter as just one example, 35% (ish) of users never tweet so we can’t listen to what’s not being said. Other social media analytics platforms (and I am biased) focus on the actions of a user and of a brand which can add to overall ‘picture’ of user/brand interaction/messages/engagement.

    Actions speak louder than words?

    The most beautifully crafted message in the world is of no value if its shared when no-one is listening. By the same token a person tweeting @ british airways outside office hours will not get a response until the next day no matter how influential they are. It’s not because the listening doesn’t work, it’s because the business has not invested in 24/7 support to engage with it.

    (note: I found this article from social listening tools!)

  10. My little contribution to this post is merely that in order to understand the adoption of social media analytics we have to understand who the staff compliment at research agencies are. The way I see it, there’s a gap between the fundamental understanding of social media and internet culture and experience as a researcher. When we have experienced researchers who have a solid foundation and knowledge base to work from and couple that with an understanding of internet culture (tumblr, reddit, 9gag, memes and blog posts) – that’s when we’ll start seeing powerful social media analytics methodologies in the market

  11. I completely agree with Cyndi De Vries. The one disconnect right now is that very few research companies have adopted social media as a part of their offerings. Social media listening is done by marketing agencies who club this as a supporting service with their core services. Most of the time clients find it convenient as it comes out cheaper for them. I think better integration of MR, Social media and Marketing activities can bring good results.

  12. I was late in spotting this stimulating post and discussion and won’t comment other than to note that most of the rebuttals to Ray’s points, including the later post by David Rabjohns, mostly consist of recycled claims regarding the hypothetical potential of social media, not what it has been actually been shown to deliver. Caveat Emptor, regardless of the methodology, new or old.

  13. I have to agree with Ray on this one. Particularly that Social Media often doesn’t answer the questions that researchers have. In the healthcare space there is a lot of good insight on how consumers talk and describe their conditons which is very rich fodder for supporting Health Literate communications. The dialogue is also rich with recommendations for treatments – natural as well as prescription and OTC therapies. The really big challenge is that consumers generally don’t talk about specific brands or experiences as they might in other industries and product categories.. This is a double edged sword of course. But due to the low engagement with specific brands or treatments, its difficult to see how social media can get the traction it may have promised early on. I’ll add too that while consumers and patients talk in open spaces, physicians and other healthcare providers generally speak only in closed spaces limiting access and the ability to really analyze sentiment. I am intrigued and see real opportunities with some of the evolving hybrid methodologies and look forward to their progression in delivering real insights and value.

  14. Having followed this debate with interest I believe (a) there is a further reason why social insight has failed to gain the traction many expected and (b) why the market research industry is missing out as a result.

    As a company that has pioneered the use of social insight we have been dismayed by the market research industry’s lack of awareness about, use of and support for statistical techniques that make social insight a very powerful research tool; and, without any doubt whatsoever, this has been a major reason why the take up of social analytics for market research has been poor. I very much hope that a forum such as this might be more open to the points I will make, but we shall see…

    The major benefit of social analytics for market research hasn’t been covered in the above debate that I can see; and if it has I’m sure I’ll be corrected! Taking Twitter as the prime example, what social insight offers is the capture of what we call the ’emotional vent’ that traditional research methods are so poor at tracking. The emotional vent is that very moment when a consumer feels an overwhelming urge to share with others — or purge themselves of — a feeling generated in response to some experience.

    By definition, emotional vents tend to be highs or lows generated in response to an appalling or wonderful experience, ranging from customer service in one’s local Tesco to the scoring of a goal by one’s local football team. The crucial points about emotional events are that they nearly always occur very close to the real-time moment of the experience, they are just as likely to be positive as negative; and they much more closely reflect consumer behaviour than the cognitive responses given in market research surveys.

    By tracking and analysing emotional vents through Twitter we have been able to make some impressive predictions at high levels of confidence. For instance, we predicted that Bob Diamond would resign from his position as CEO of Barclays over the Libor scandal while everybody else was saying he would survive. We also predicted that Morrisons would have problems significantly greater than any other grocery retailer before anybody else; and we still maintain that the company’s problems run much deeper than the lack of an online service or local stores to which the business’s problems have been attributed by its CEO and leading industry analysts. You don’t have to take our word for this — we alone tweeted about Bob Diamond’s demise before it happened and we blogged about Morrisons before its problems surfaced.

    Of course we might just have “got lucky”, but we know this wasn’t the case. Why? Because we could track the emotional vents of consumers that were bound to lead to the conclusions we predicted.

    This raises the inevitable question of how we can measure the impact of emotional vents, which leads me to the lack of knowledge among the market research industry of the statistical techniques mentioned above. The technique in question is bootstrapping, or resampling, which despite its simplicity and power remains a dark art to the vast majority of market research professionals.

    The point about social media is that they are NOT a representative sample of the UK population, but what the market research industry needs to understand is that this can be a good thing just as much as it can be a bad one. Why? Because people who feel strongly enough to post an ’emotional vent’ are typically those who fall into the “outliers” box with traditional sampling methods (it is simply incorrect to say that people only use social media to complain. Just as many use it to say nice things about companies and brands).

    I will conclude there but would be very happy to continue to debate these points if others would like to do so. The bottom line is that traditional market research methods are failing to work as well as they once did as the population fragments, consumers are now so over researched and, ironically perhaps, because of social media.

    The shame is that companies and brands are missing out on crucial insights that traditional methods are failing to spot in time or simply missing all together. As a result the market research profession is also missing out on opportunities to move the industry forward because it is failing to embrace the techniques and innovation that could address these problems.

  15. Hello Ray,

    We couldn’t agree with you more on this subject.
    “Social networks are where you get to be spontaneous as a brand, you can engage directly with your clients. But it’s not a replacement of current research-methods, rather a necessary extra.”
    This inspired me to write a post about the subject myself.
    You can read it on our blog here:

    Keep inspiring!

Join the conversation