Download the

Feelings. Nothing More Than Feelings.


By Dr. Stephen Needel

Feelings, a 1974 song by Morris Albert, might be the worst song ever to hit the charts (at least for males in the early 70s). A recent Quirk’s article ( about feelings may not be the worst article they’ve ever published, but it ranks up near the top. The author is flogging a new voice analysis system that claims to detect the passion in response to a new product concept and enables better performance forecasts. Full disclosure – what I know about audio engineering and sentiment analysis would fit in a cocktail glass and leave plenty of room for my martini. The good news – you don’t need to know anything about either of these topics to appreciate the lack of empirical evidence in this article.

The article starts with, “Fundamentally, consumers adopt new products and services that improve their lives.” This is by no means fundamental; neither my new Smucker’s Chocolate Coconut Ice Cream Topping nor my new Gia Russa Bolognese Pasta Sauce is likely to improve my life, although they both taste good. Most things in our pantries are not life-altering, and food is what we all buy most of on a transactional basis.

The author believes the “enormous question on the table…is how to identify product concepts that have high probabilities of building deep emotional connections with consumers.” No, the enormous question on the table, from a new product forecasting point of view, is how to better predict new product trial; that may or may not involve emotions. “Consumers do not talk about products using multi-point scales”, the author claims, “They are unnatural modes of expression.” Consumers don’t talk much about products at all unless we ask them – they have better things to do with their lives (except those who live their lives on Facebook). A well-constructed set of scalar items is no less sensitive or less informative than the open-ended questions they would have us use. Neither modality is more or less natural to consumers.

In a comparative test, “the language-based sentiment metric [open end] yielded a coefficient of variation that was five times higher than the scalar method.” As if this is a good thing. What they obtained was a highly variable metric relative to scales, and that’s not very good for a prediction exercise.  By the way, we do not expect much variation from a 5-point scale either, unless the product is particularly polarizing. And we won’t mention the issues of a COV with a non-ratio scale or the wisdom of asking 35 purchase intent questions at one time.

In lauding their voice analysis method rather than the typing-an-open-end-answer, success for the former is claimed because respondents produced 83 words per stimulus compared to the typed 14 words. I can write a scathing review of a restaurant in lots of words or I can just say “It Sucked!!!!” I’m pretty sure the latter conveys exactly how I feel about the restaurant and why you shouldn’t go there. Word counts are a ridiculous measure of anything.

There’s a lot of fake math in this article surrounding the audio sampling rate – just ignore it, it’s wrong.

Using voice analysis, the author reconfigures the maximum trial potential of a product as the subset of respondents who (a) express positive sentiment towards the concept, (b) show positive activation (an expressed desire to do something), and (c) do so in a passionate way (as defined by the voice analysis). Spoiler alert – there is no validation for this assertion. I’m not saying we, as an industry, are great at predicting new product success. I do, however, expect someone who’s claiming to have a better way to do this to show us some data to back up the claim.

I’m sure this company wants to know what I’m feeling about their new voice analysis approach. I’m not a likely adopter, because while I experienced positive activation in a passionate way, my sentiment is anything but positive.

Please share...

6 responses to “Feelings. Nothing More Than Feelings.

  1. I doubt Justin even read your article Steve. Just more advertising under the guise of useful comment. I was also puzzled by the use of COV, since it is only applicable to ratio data and the metric indicates spread of variance. So not sure what was meant by being five times better than the COV from ordinal data. Despite your heavy criticisms it is clear that technology like this and VoxPopMe based on capturing and encoding text data derived from audio is very insightful. Just based on the quality of text and sentiment analysis, this is a promising research direction.

    Tom Anderson of ODINText recently asked for survey participants to comment on use of open-endeds and the findings were dismal in the extreme. The tail (avoiding the need to analyze) is clearly wagging that research survey dog (lets drop open-endeds). This is a sad indictment of the industry, largely due to the “everything online” mentality. I would need more validation of the audio analysis technology before claiming it is predictive. N=300 is hardly enough to draw conclusions, but i remain optimistic.

  2. Hi Chris – I have no problem with text analytics – I’ve seen Tom (among others) do some very good stuff. I have a problem with the assumption that researchers aren’t smart enough to figure out whether someone holds a strong or weak opinion. I don’t think we need to go to audio analysis (or, for that matter, any other neurophysiological measure) to determine strength of opinion.

  3. Thanks for this, Steve. Having worked for many years in an R&D capacity, including close involvement with STMs, I share many of your thoughts. Substandard practice is easy to beat.

    I would be interested in hearing how this new methodology stacked up against neuro, facial and other “implicit” methods.

  4. As with any organization selling anything (widgets or market research services) the assessment of reliability and validity is superseded by your organization’s goal to generate revenue.

  5. Pingback: Plunging Blog

Join the conversation