Editor’s Note: Ray Poynter’s recent post “Why Has Social Media Analytics Met With Limited Success In Market Research?” continues to stir up a much needed debate in the industry about the role and impact of social media analytics in market research. David Rabjohns rebutted here, and today we have Rob Key, CEO of social intelligence consultancy Converseon, going even deeper. It’s good stuff indeed.
Both Ray and Rob will be at IIeX in Atlanta next month, and although we don’t have a debate planned on stage, I bet there will be some really interesting evening discussions happening!
By Rob Key
In a recent editorial, Ray Poynter recently raised some interesting reasons why he believes social research has, to date, not had more impact on market research. As the founder of a leading social intelligence consultancy who has lived through the evolution of social research for over a decade, I continue to see misconceptions about what listening is, and is not, and its role in market research.
The truth is social listening has already been through several phases of evolution and rapidly entering the next one. Like many other technologies, the category was not immune to the Gartner Hype Model for technology innovation. In the model, a trigger technology leads to a peak of inflated expectations that are simply not realistic, which then leads to a trough of disillusionment before a slope of enlightenment kicks in where the evolution of the technology grows to the point that it begins to fulfill its potential.
In 2006-07, social listening began its hype phase when relatively simple do-it-yourself dashboards, with limited data functionality and generally driven by the need for communication teams to “monitor the brand,” began to become pervasive.
In 2008-2011, as social listening and intelligence became de rigueur with the boom of social media growth, there began to become greater demands on the technology as primary voice of customer input into many functional areas in organizations. This included those areas, like market research, that required high levels of data accuracy and reliability even those these platforms were not designed for this kind of use.
In 2012-13, there came the inevitable disillusionment phase for some brands that had greater, and perhaps unrealistic, expectations and hopes for basic social listening technology as they perhaps hoped that the data aggregation nature of these tools could also fulfill the data intelligence needs too – not realizing they were separate technologies. They began to ask themselves, “what now?”
But we’re now entering a new era. The end of 2013-14 has marked the beginning of the “slope of enlightenment” phase where the technologies and approaches have matured to the point of being able to successfully meet the high hopes and expectations of the marketplace in ways not previously possible. It’s in this period where we will see even greater adoption, which I expect, in turn, to drive even greater transformation.
By way of context, it is important to note there is a significant difference between what is often referred to as “social listening” and “conversation mining” or true “social intelligence.” The former simply aggregates data from multiple sources, utilizes some very standard – and often quite primitive — text analytics with high level of imprecision and unreliability and allows users to query the data through equally primitive “Boolean” queries, or word constructs designed to try to isolate specific datasets. Much of the criticism regarding social listening from the market research community stems directly from trying to retrofit these basic monitoring tools for market research purposes since that’s not what most were designed for. They were designed to monitor, in near real time, general conversations about ones product or brand and getting a general indication of what is happening in the market, generally for campaign and crisis management.
Some researchers believed that the easy-to-use, low cost nature of these solutions could be useful to provide rapid research findings. However, using just these tools for advanced research and business intelligence purposes is not only challenging, but can be dangerous. Incomplete data, limited metric options, often inaccurate sentiment analysis and very high levels of irrelevant records due to the limited powers of Boolean queries can lead many researchers to exactly the wrong conclusions. Making business decisions on this analysis is clearly problematic. When a report derived from a basic monitoring tool was provided to the CEO of one of the world’s leading technology companies, the CEO quickly read the small print which said “sentiment is estimated at 60 percent,” before turning to the team at hand and asking, “if this is true, then why are we even having this discussion?” This certainly represents a trough of disappointment.
To deal with these inaccuracy issue, some market research teams and vendors had to turn to interns or other dubious processes to hand code the data for sentiment. But this too is highly problematic. Humans often disagree with each other. A sampled data set hand coded by human is also equally distorted – some agree at less than 60 percent. At my firm, we have to have three independent trained “coders” review a record and if there is a disagreement, we have an adjudication process. It can become quite complex, especially dealing with large data sets and with sensitive metrics, like emotion. And it’s simply impossible to do at scale. On the other extreme, basic “automated” sentiment based on a simple rules based approach – the approach used by the vast majority of social listening tools — simply can’t capture sarcasm, slang and the other nuances of language that make up much of social conversation with any level of reliability. Sentiment scoring using this methodology is also often no better than 60 percent.
Add to this the challenge some researchers have had in terms of trying to correlate social data with other sets of analysis, such as customer satisfaction, product sales and more, it’s no wonder that the technology hit the trough of disappointment for some. It seemed not representative, or even misleading, at times.
But to those who truly understood the technology, this was no surprise and, in fact, was inevitable. Pretty looking charts in off-the-shelf dashboards hid the significant data quality issues that were pervasive, and remain so in many areas, and it was just a matter of time that these limitations shined through. Garbage in, garbage out certainly applied here.
Enter the Age of Enlightenment.
The good news is that there are now a new generation of solutions and methodologies that address the limitations often cited by critics and were inherent in many basic monitoring solutions and provide researchers with advanced capabilities that address almost all of the perceived limitations.
The ability for algorithms to effectively analyze human language has grown leaps and bounds over the last 18 months. Instead of simple “rules-based” approaches, the newest techniques also embrace machine and “deep” learning methods that continually evolve based on large-scale human training. The best of this technology can also be “tuned” to specific brands and categories so that it understands the word “small,” for example is good if you’re selling smartphones, but not so great if you’re selling hotel rooms. Semi-supervision methods can allow “low confidence” records to get reviewed by humans and in turn continually improve the system. This is why the newest sentiment analysis finally is achieving near human level precision at the speed and scale that only software can provide. Combining this analysis with the power to “backcast” – or go back in time to a precise moment to capture the conversation exactly as it was then – is especially unique to social research and quite powerful.
Instead of a one size fits all approach, these new language technologies also allow researchers to break out of the confines of limited metrics and include emotion, intensity, intent, and the ability for researchers to build an almost unlimited number of their own custom classifiers without having to use primitive Boolean queries. This allows for the segmentation of data in much more meaningful ways – such as classifying the data based on where they are in the buyers journey (pre versus post purchase). Machine learning approaches have superseded Booleans with much greater efficiency and reliability. In a recent test we have conducted to try to isolate the product “spam” (the food) from the rest of the online “spam” conversation, the best boolean queries were only able to generate a 15 percent relevancy rate in the data, while machine learning rapidly achieved over 90 percent.
These advances are helping to provide much greater insights, more rapidly and reliably, than ever before. While previous approaches made it difficult to attribute conversations to a particular group or segment, we can now do so with great precision. We can map conversations to specific segments, identify not just gender and basic demographic information, like location, but also create a hierarchy of interests, psychographic attributes and discover unmet needs and more. We now know not just “what” is being said, but who is saying it and why.
Highly accurate data, the ability for researchers to easily custom classify data to help answer a wider range of research questions, the ability to apply the analysis to people and segments and integrate this data with other datasets for correlation are clearly transforming the industry.
But can it be predictive? We are finding that the answer is a firm “yes,” as long as the data and methodology is solid.
In one recent experiment, we took a year of historical conversation regarding a certain car brand, enriched it for high accuracy and built custom classifies to isolate the conversation into pre and post purchase to see if higher positive sentiment led to more car sales. What we found was that higher negative conversation lead to less car sales.
Another case study, conducted in partnership with Professor Wendy Moe at the University of Maryland, for a major enterprise software brand, tried to correlate social conversation – especially sentiment – with offline brand tracking. A simple average using regression analysis showed almost no correlation (.002), but by cleansing the conversation of offers, advertising and other clutter, and applying weighting to the conversation based on influencer and venue, the researchers were able to find a strong correlation of .604. http://www.msi.org/reports/social-media-intelligence-measuring-brand-sentiment-from-online-conversatio/
This is a powerful example of why decomposing the social conversation through advanced filtering and applying effective methodologies that go far beyond the capabilities of a basic monitoring, and avoid the pitfalls of simple average sentiment approaches.
Social research and intelligence is clearly powerful, but not a panacea. There will of course always been room and need for additional “prompted” approaches to get to insights that go beyond social. That’s why we believe a triangulated approach to some research challenges is essential. I also believe we will see nano-survey approaches in combination with social research, as well as other techniques. And given that there are now many flavors of social listening and intelligence and they touch so many areas, success will require brands to develop a comprehensive approach to determine exactly where and how social intelligence can and should support an overall market research strategy.
The convergence of “big data” technologies, a new generation of linguistic analysis and filtering and the application of new, proven science-driven methodologies are true game-changers. And they are already being applied by leading brands to research needs as wide-ranging as competitive analysis, campaign planning, brand tracking, product development, and much more. By combining social conversation data with search and other data, one can map the customer journey and discover perceptions along the way, gaps and unmet needs. Further, by being able to listen to certain segments, we are now able to create passive “panel” listening to key groups for ongoing, meaningful insights, and then find “look alikes.” And this is just scratching the surface of where this is going.
Of course, all change is difficult and early versions of technologies always take longer to mature than we hope or anticipate. And the broader adoption of social research and intelligence will require the industry’s willingness to reconsider current models, evaluating the newest breakthroughs with fresh eyes and challenging some notions of what it can, and cannot do. Social intelligence is now proving to be both qualitative and quantitative. But like most good things in life, it turns out that the best of social research and intelligence takes some work and modeling that goes beyond basic monitoring dashboards. And that’s how it should be. Humans are still needed to connect the dots while technology is finally at the point to do the heavy lifting more efficiently and reliability.
The benefits of social research and intelligence are powerful, growing rapidly and has already become truly transformative to those who embrace it with an enlightened approach.