Garbage In, Garbage Out

Why should companies be worried about conducting scientifically sound, replicable studies when garbage “research” gets such good coverage in USA Today?

“Survey:  Over half of fliers don’t want to sit next to kids on planes,” screams the USA Today headline from August 26.  And it’s not just a blurb, but a multi-column article taking up about 20% of the page.

Problem is, it’s not until the fifth paragraph of the story that a passing reference is made to the fact that this is meaningless “data” – it was simply a poll some travel website posted up on their home page.  Out of 21 column inches of copy, headline, and graphic devoted to the story, we read this in reference to the methodology:  “Skyscanner spokeswoman Mary Porter says results of the unscientific poll are not surprising.”

Unscientific poll?  Apparently that’s journalistic slang for “complete nonsense.”  One of many travel-comparison websites throws a poll up, and 2,000 people who happen to be visiting that site respond.  Representative sample?  Bah – not necessary.  Statistical significance?  Don’t bother.  Replicable data?  Who cares?  It’s a poll!  It was on a website!  There must be a story in here somewhere!

I’m not criticizing Skyscanner.  Polls on websites are done all the time.  Out of curiosity, I clicked on a few different sites.  NFL.com, ESPN.com, ProFootballWeekly.com (can you tell I’m a football fan?), CNN.com, ABC.com, and many others feature “polls” prominently displayed on their home pages.  And even though their “research” is complete drivel, it got them 20% of a page in USA Today (and who knows where else).  That’s priceless publicity.

But seriously, this is news?  USA Today should be ashamed of itself for creating an entire story around something with no more validity than if Skyscanner had made up the data out of thin air (in a bit of irony, the story’s associated graph even reported the findings with decimal points, as in “Yes – I don’t have young children, want to sit as far away as possible:  53.3%” – maybe reporting the decimal point makes it sound more credible).

The amazing thing is that apparently the article’s author, Gary Stoller, and his editors did nothing to figure out even what kind of people responded to the poll.  Are these mostly men?  Young people?  Frequent fliers?  Budget travelers?  Business travelers?  International travelers?  Heck, are these even Americans?  Eh, why bother – they’ve got 2,000 people’s opinions and we can write a story about it!

Not only did USA Today do a major disservice to its readers by treating these “findings” as valid research, but it did a major disservice to the entire research industry.  We’re regularly driving ourselves nuts trying to figure out things such as whether using targeted phone sample or an online panel will provide more accurate, projectable data.  Stupid us – all we had to do was throw a poll onto some website, get 2,000 people to respond, and we’d have our data!

Shame on you, USA Today.  And shame on every media outlet that reports “polls” like this as having some relation to reality.  And shame on every corporation that believes they can interview people at a trade show, or throw a poll up on their website, or e-mail a Survey Monkey link to a couple hundred customers, and suddenly they have valid research.  Because let’s be honest here – the news media aren’t the only ones paying attention to garbage “research” like this.