Generalizing: The Bane of Insights

 

I often wonder whether, in research, we spend so much time navigating the complexities of gathering the data that we neglect the all-important field of communicating what we find.  Issues such as online representativeness, phone response rates, and newer forms of data collection (mobile MR, social media sampling, etc.) take up so much of our mental bandwidth that it can be easy to give short shrift to clarity and accuracy in reporting.

One of the biggest and most potentially toxic issues is generalizing.  Marketers dream about homogeneous populations – segments composed of consumers who are all looking to buy a new minivan, or who all have price as the number one criterion when choosing a cell phone provider.  Because of the lure of homogeneity, it’s very tempting to generalize a segment that shows a greater proportion of certain people as being comprised solely of those people.

Geodemographic clustering falls prey to this quite easily.  When I first learned about this technique a couple of decades ago, I was initially quite impressed that companies could identify clusters of people who were all “upscale Caucasians who are early adopters of technology.”  It was a huge disappointment to find out that this segment, rather than being exclusively comprised of these people, simply contained 20% of these people, rather than the 8% who could be found in the general population (I’m making these numbers up).  Although many purveyors of clustering clearly identify their methodology and how the technique is built, I’ve seen how this process is often used by marketers and researchers.  Rather than discuss a cluster with a higher proportion of the desired target, they discuss the cluster as containing nothing but the desired target.

Research reports do this far too often, as well.  For instance, in a major study of charitable donors for the Russ Reid Company (http://www.greymatterresearch.com/index_files/Heart_of_the_Donor.htm), Grey Matter Research found that donors under age 40 are more likely than older donors to have considered supporting a new (to them) organization for the first time in the last year (51% to 38%).

Now, this is a statistically significant difference that can be critical to marketers.  Yet depending on the wording of the report, it could be very easy to lead readers to the feeling that all younger donors out there are running around checking out new organizations to support, while older donors don’t do this at all.

We see this all the time in real life:  Democrats support same-sex marriages.  Evangelical Christians are right-wingers.  Engineers are socially awkward.  Wealthy people own very expensive cars.  Young people aren’t brand loyal.  These are called stereotypes, and while they sometimes can reflect tendencies (certainly a wealthy person is more likely to own a Bentley than I am), they don’t hold water for every member of a group – sometimes not even for a majority of a group.

Let’s face it – it’s much easier to sit in front of a group of executives and explain, “Latinos simply aren’t interested in your product, while Asian-Americans love it” than it is to provide all the statistical details, differences in the data, etc.  And sometimes marketing strategy can be as concise as saying, “Let’s pull resources from our Latino marketing and put them toward reaching the Asian-American market.”  But it’s dangerous in reporting to over-generalize, or to lead the researcher user to believe that 25% versus 16% means one segment loves the product while the other doesn’t.  Even when the generalization happens unintentionally, it still harms the effective communication (and therefore the effective use) of the research.

 

Photo by Kelly Sikkema on Unsplash