SIS INTERNATIONAL RESEARCH: NYC FOCUS GROUP FACILITY
Our new GreenBook Directory site is live!
Your #1 strategic guide to consumer insights.
Qualtrics X4 The Experience Mgmt Summit
Brand & Retailer tickets for all IIeX events now start at just $99! Get or give one today!

Have The Statisticians Have Taken Over The Segmentation Asylum? No.

Over twenty years of conducting segmentation studies using this powerful combination of tools for marketing researchers is not at all a recipe for disaster. Is this combination to be used all of the time? Of course not.

one-flew-feature

 

By Steve Cohen

It’s not often that I take offense at something written on the Internet.  After all, it’s a wild, wild West out there where moderation is often in short supply and signs of intelligent life are hard to find.

With this in mind, during a recent Google search I came across a White Paper that claims to “debunk” the use of “fancy, schmancy” segmentation procedures.  In particular, the author laments the fact that the statisticians who “are running the asylum” recommend for segmentation studies the use of MaxDiff Scaling and Latent Clustering (which is called Latent Class or Mixture Models by all people I know).  In fact, the author states that using both in segmentation studies is a “recipe for disaster.”

Wow.  Just wow.

As someone who has won several awards for my work introducing and using MaxDiff and Latent Class segmentation in the marketing research community, I had to read this document in depth.  When I did, I took immediate offense at several of the boneheaded assertions in it.

Let’s take a brief tour of what the author claims.  First, MaxDiff

“… is a great measurement tool that should not be used as the source of segmentation inputs. Sound segmentation inputs need to be measured at the individual level and use a method that can be readily reproduced in “short forms” applied during follow-up research.  MaxDiff does neither.”

I have two very serious problems with this.   First, in my experience using MaxDiff in segmentation studies since 1997, I know that MaxDiff can produce stable, reliable, and very usable results that provide much better differentiation and  interpretability than traditional methods.   And, second, I contend that a clever analyst can develop a very compact and accurate short-form using MaxDiff results that can be applied in follow-up research.

Heterogeneity

MaxDiff does measure segmentation inputs at the individual level.  These measures are the responses collected in the best-worst choice tasks.  Under certain circumstances, what MaxDiff can do is yield individual-level utilities that are estimated using a Hierarchical Bayesian multinomial logit (HB-MNL) model.

The author seems to be blissfully unaware of the discussions in the marketing science literature these past few years about the nature of segments.  Prof. Greg Allenby of Ohio State has argued that heterogeneity (segments) should be measured on a person-by-person basis and we should think of segments as people who behave at the extremes, based on an examination of the individual-level utilities.  Others, like Michel Wedel at Maryland and Wagner Kamakura at Rice, claim that segments are really constructs that help managers deal with the complexity of markets by providing shorthand ways of talking about consumers and customers in aggregates — which we call segments.

My own view leans heavily to not using individual-level utilities estimated with hierarchical Bayesian tools since the utilities are assumed to be drawn from a normal distribution  — meaning the distribution of utilities is smooth and thus does not display any obvious places to “cut” into groups.  What I do instead is use Latent Class Models, which assume that the utilities can be estimated to be lumpy and multi-modal – meaning that segments, if they exist, can be discovered.

By the way, since Choice-Based Conjoint Analysis also uses choice inputs and then estimates individual-level utilities using HB-MNL, would the author make the same argument to debunk CBCA?  Somehow, I think not.

My guess is that the author has been using Sawtooth Software, which does generate individual-level utilities, in a rote way too often and has not paid much attention to the behavioral science behind segmentation nor to the assumptions underlying these tools.

Short Form MaxDiff?

Let’s examine the second claim that MaxDiff does not yield a method that can be used in a short-form after the segmentation study is complete.  Specifically, the author says,

“There is no way to reproduce the MaxDiff importance scores in a short-form classification algorithm.”

First of all, follow-up short-form classification surveys are never be designed to reproduce the MaxDiff importance scores.  What is this claim all about?

Rather, as in traditional segmentation studies which employ Discriminant Analysis for post hoc classification, the function of the short-form is to assign people to known segments which have known characteristics by using as few questions as is reasonably possible.  Got that?  We are not looking to reproduce importances, but just to put people into groups with good accuracy.

I find it hilarious that this wrong-headed assertion is compounded by this declaration about short-form classification tools:

“… the accuracy rates are so low they would scare you.  As a result, short forms generated off MaxDiff segmentation schemes tend to be both lengthy and inaccurate.”

I can state categorically that, in my experience, we can create such short forms which are as accurate, or even more so, than traditional methods and are much more compact than traditional methods.  I have personally created such short-forms and these contain typically less than 10 questions with accuracy rates in excess of 85%.

Latent Clustering (sic)

Latent Class (LCM) or Mixture Models are based on sound statistical foundations and have a long history of use in marketing science and many other disciplines for uncovering hidden (latent) groups (classes or segments).

So what is the author’s beef with Latent Clustering (sic)?  Again, I quote:

“Consumer segmentations are generally done on survey data and respondents have the unfortunate tendency to use scales in slightly different ways from each other (see benefits of MaxDiff). The reason this is a problem in Latent Clustering is that frequently the model tends to form segments based on how people use the scale (e.g., high raters or middle raters) rather than what people were trying to tell us on the scale.”

Hello?  Respondents using a rating scale badly is a ubiquitous problem, not only for clustering or grouping of any flavor, but also for brand ratings and many other typical marketing research tasks.  Blaming LCMs for how people answer surveys in a biased way is just absurd.

Is there a suggested alternative?

“Transformations (e.g., within-respondent-standardization) that are an effective solution to this issue in Euclidean distance models do not prevent Latent Clustering from generating these meaningless groups,”

I really tried to untangle this word salad, but there are so many ideas happening in this one sentence, I was forced to reach for the aspirin bottle.

But suppose just for example that there are some survey respondents with no or little within-person variation.  Claiming that the within-respondent standardization supposedly solves this issue is wrong; it can create yet another set of thorny problems.  Think about it.  If a respondent “straight-lines” a series of survey attitudes (which happens quite frequently), a within-respondent standardization will require dividing the mean response for each person by his/her own standard deviation, which is exactly equal to or very close to zero.  Good luck with that being an effective solution.

Mixed Levels of Measurement

Yet another beef with LCMs!

“The ability to mix metrics generates the temptation to throw in the kitchen sink and segment on virtually the entire survey (attitudes, needs, behaviors, demographics and even brand usage!).”

Good lord!  You mean to say that there are researchers in our industry who dump the kitchen sink in a segmentation analysis without even thinking about what they are doing?  Oh, no!  Where have I been all these years?

My contention is that, used judiciously and wisely, variables at mixed levels of measurement are a great help in developing actionable segmentation solutions.  Dumping everything in at once is not a flaw of LC models, but rather of an ineffective analyst.

So what is the suggested alternative?

You dear readers who have actually spent the time to read the quoted article were, no doubt, eager to hear the punch line.

Once the author has “debunked” these tools, surely the magic bullet, the keys to the kingdom, the secrets of life, and the sacred tablets as written by the author will be shown to us lowly mortals.

And what do we get?  What do we hear? What is the long-awaited wisdom?  What should we do instead of using these heinous methods?

(That is the sound of crickets.)

Summary

I suggest that this author clearly needs to get a firm grip on the behavioral and statistical assumptions, theories, and methods of MaxDiff, Latent Class Models, and Hierarchical Bayesian modeling.  Spending time trashing these modern advances, misunderstanding their uses and application, and then suggesting nothing to replace them is not even remotely helpful.

Expecting everyone in marketing research to be an above-average analyst born in Lake Wobegon is foolhardy.  Perhaps the author will come to realize that some people are just good examples of the Dunning-Krueger effect.

Over twenty years of working with these tools have convinced me that conducting segmentation studies using MaxDiff and Latent Class models represents a powerful combination of tools for marketing researchers and is not at all a recipe for disaster.  Is this combination to be used all of the time?  Of course not.  Marketing researchers should select the best methods and statistical procedures to meet the objectives at hand.

Are the statisticians running the segmentation asylum?  Hardly.

Let’s not follow flawed guidance that may not be based on a full picture of the collective experience and best thinking of many experts (not just me!).  Otherwise, the incompetents may end up running the segmentation asylum and that is really why it could get scary out there.

Please share...

9 responses to “Have The Statisticians Have Taken Over The Segmentation Asylum? No.

  1. Perfectly done Steve. I wholeheartedly agree on every point. MaxDiff provides a wonderful basis for segmentation, especially in international studies where you want both variance and to minimizes cultural bias in scales. And LCMs almost always (in my experience, using multiple methods on the same data) find the best solutions. With “best” being a combination of mathematical elegance and deep, rich profiles beyond the basis variables. Perfection and segmentation is a Santa-Claus like myth; categorizing markets of millions of individuals or businesses is always a gross oversimplification of reality. But the fact is, when designed, analyzed and evangelized well, segmentation can materially improve your odds of success when marketing to, or developing products for segments. And that’s what businesses really care about.

    My guess is the author has (in addition to crickets) a pretty narrow point of view on a segmentation approach. I prefer to let the data reveal the story, through thoroughly investigating multiple basis variable sets, and often using multiple techniques. The key is really vetting the resulting profiles against the decisions the client business is hoping to make. Otherwise, why do it at all?

    Anyway, thanks a million for doing a wonderful job slaying a myth in the making.

  2. Thanks for this! I also find it discouraging how little segmentation has actually progressed and how much baloney is still being written about it. In 2015, to many marketing researchers and clients, segmentation still comes in one of two flavors: 1) cross tabs with (often very arbitrary) a priori segments as a banner; and 2) k-means clustering of principal components (“factor”) scores that have been derived from ratings of a long list of (often improvised) attitude statements. Then we’re told segmentation “doesn’t work”…though many of the complainers keep repeating their mistakes again and again. How will we ever be able to handle Big Data?

  3. I like Steve’s comment. However, there is one thing that continues to bother me. That one thing is the nature of segmentation. What do the segments really mean?

    1. Do they represent operationally USEFUL groupings of people, e.g., to a marketer?
    2. Do they represent TRULY BASIC GROUPS in the population, with the possibility that these groupings may lead to more insights?

    I have been struggling with these problems for almost 50 years. I came upon the issue in my introductory reading for the PhD, when I was learning about the ‘four basic tastes,’ now growing to five or six, but then four. Were these tastes really unique, biologically different dimensions of taste, corresponding to real divisions in the way we sense stuff on our tongues? Or were these tastes nodal points in an cloud of different sensations, and that the underlying reality was really the cloud, not the nodal points? If the latter be true, then scientists were simply trying to make some sense by clustering together a continuum, and by so doing, deal with an otherwise difficult to understand reality.

    Steve alludes to that point. The personal reality for me as a scientist is NOT the statistical and computational machinery, but what one does with the segmentation. Does the segmentation point to mental ‘primaries,’ opening up whole new vistas of understanding? Or does segmentation devolve down to a way to make some sense of what is otherwise a continuum without form? Are the segments simply like those familiar names we assign to images that we think clouds make, because of our very human need to impose order.

    There is an ulterior motive. (There always is). I personally am working on a new science, Mind Genomics, which hypothesizes that in every area of human thinking, in every topic where ‘man is the measure of all things,’ there are different ways of looking at the same facts, the same messages. These different ways are the mind-set segments. In the end, will the mind-set segments allow us to ‘sequence a person’s mind’ in terms of what the person holds dear in different topic areas (insurance, breakfast, reaction to children, etc.)? And can we link the array of mind-sets of a person in those different topic areas to either genetics on the one hand (think 23 and ME), or database information (think Experian information)?

  4. Segmentation is really useful and should be used much more. Just something to bear in mind though that it’s more important to have actual ‘real’ segments (where you know real people that fit in each segment) than what type of statistics you use. And segmentation is only useful if you can implement the segments and treat them differently. If you can’t and still use your shotgun don’t waste your money and time.

  5. Related to Howard’s post, at what point do we start to ask what meaningfulness we get from segmentation that requires “10 variables” to deliver segmentation? Can that be telling us something, that maybe the segments obtained are stretching the maths?

  6. @Chris – I look at it from the other end – the meaningfulness of a segmentation is, to me, the utility it provides in marketing. This perspective suggests ontological status is irrelevant. If the segments one derives are (a) stable and (b) target-able, the segmentation may have utility. Of course, this takes it out of the realm of an “insight” and into the realm of a statistical pattern, but if that pattern is useful, why not?

  7. I’d like to build on Howard’s comment. The number one problem with segmentation is marketing actionability. In particular, I would like us to ask marketers how their customer segmentation has led to improved marketing productivity. If there is no proof either the segmentation was ultimately empty calories or it represents sloppy follow through. To drive action in a digital age, marketers must do a few things: 1) integrate digital profiling variables into your scoring and scaling models so you can actually deliver ad impressions to the right person with the right message at scale, 2) segment moments which define relevance better than assuming persistent interests on the part of a consumer and 3) prove that what you did actually improved advertising productivity. about 2 years ago I blogged about this and it was my most shared blog ever…over 3,000 shares, indicating the frustration that marketers have with current segmentation practice. http://blog.joelrubinson.net/2013/04/four-new-approaches-to-consumer-segmentation-in-a-digital-and-social-age/

  8. I want to thank the commenters so far for the cogent remarks and I would like to make a few points in return.

    First of all, we all agree that the championing, implementation, and usage of segmentation results do make or break its utility. However, I daresay that this feature (bug?) is not endemic to segmentation studies, but to any initiative undertaken by a firm. Many efforts, from CRM to price changes to ad campaigns, can fail to achieve their goals if they are not well conceived and executed.

    Second, my goal in the post above was not to enter into this discussion of implementation, but rather to point out something that Kevin Gray puts so well. Bad questions and outdated analytic tools can doom a segmentation study all by themselves. Phipps Arabie, at that time the President of the Classification Society of North America, wrote a review article in 1994 on Cluster Analysis in the book, Advanced Methods of Marketing Research, in which he said, “Tandem clustering (i.e. factor analysis followed by cluster analysis) is an out-moded and statistically insupportable practice.” Twenty years later, we still find many textbooks, university classes, and analytic consultants which still hold this practice in high regard. Get with the program folks!

    Finally, some commenters question whether segments do in fact exist. I take my cue from the wonderful text book, Market Segmentation: Conceptual and Methodological Foundations, by Mickel Wedel and Wagner Kamakura. They write, “The development of segmented marketing strategies depends on the current market structure as perceived by managers.” Segments are really a shorthand way of describing customers. Given how complex the world is, the grouping of consumers into segments is really a clever, yet simple, way for managers to approach markets. Obviously if these groups hold no relationship to any reality, any strategy or tactics based on them will fail.

    Yet, I also want to reiterate that is really does make a difference how one does conducts the analysis of segmentation data. Again, Wedel and Kamakura, “The identification of market segments is highly dependent on the variables and methods used to define them.” I hope that my blog post has made it clear that variables and methods do matter — a lot.

Join the conversation

Steve Cohen

Steve Cohen

Partner & Co Founder, In4mation Insights