Civicom: Your project success is our number one priority

Is Market Research Reliable?

How reliable are market research results to normal people? Findings from Morpace's May 2012 omnibus survey of 1,019 U.S. respondents help shed some light onto this topic.


By Greg Deinzer

The October 23rd posting by Dr. Bruce Isaacson entitledThe Quantity of Presidential Polls and the Quality of Marketing Research started me thinking again about the perceived reliability of market research results.  I highlighted the word perceived because as an honest MR professional I am confident that 99 and 44/100% of us in the business try hard to ensure that best practices are effectively implemented on every study.  Or, at least we try hard to obtain an ending sample size greater than one – even if that one respondent is your mother.

So, assuming that the methodology, sample size, numbers, significance testing, etc. are pure from a market researcher’s point-of-view, I wonder how reliable market research results are to normal people.  Findings from Morpace’s May 2012 omnibus survey of 1,019 U.S. respondents help shed some light onto this topic.


When asked about perception of market research results overall, 24% answered 6 or 7 on a 7-point scale (where 1 means very unreliable and 7 means very reliable).  Do you sense a little skepticism amongst the general population?

Interestingly, perceptions change dramatically when market research is attributed to various sources.  Research credited to a scientific journal, for example, is seen to be much more reliable than that coming from a political organization.

What implications does this have?  I’m not quite sure. One thing I am certain of is that I’m glad I work for a market research company. It’s encouraging to know that nearly one-third of the people who read my reports don’t think I’m lying to them.

Please share...

17 responses to “Is Market Research Reliable?

  1. I’m with you Greg – I’m not sure why we should care either. I would also think reliable is the wrong metric. Maybe believable? credible? But again, why would we care what “normal” people think about this? Aside from we want them to participate and believe their participation is truly useful.

  2. Interesting – coming 3rd only after Scientific Journals/ Universities can’t be too bad. I think the source attribution is critical for credibility, perhaps regardless of the depth of knowledge of that organisation. Be interested if this were tracked over time, what happens as more and more “Research” gets published more and more broadly.

  3. I wonder if there is some confusion here regarding “reliability” vs. “validity.” Market research often doesn’t get much attention among the general population unless the result is “surprising,” which makes it newsworthy. That’s a validity issue, not a reliability issue and I’m confident that most non-researchers don’t know the difference. The non-researcher is as likely to say that the result is unreliable as to say it is invalid because he or she doesn’t know the difference. Based on my 33 years of experience as a statistician in market research, I’d say there are more reasons to question the validity of the research than the reliability of it.

  4. I am not surprised that only a third of “normal” people or the general public feel that MR is reliable. MR has such a mxed public image due to poor political polls – Push-Pull polls, candidate polls to raise money, etc. – and the fact the media often focus on inaccurate product polls (e.g. New Coke), or law suits focusing on government studies. It is a wonder that anyone believes in polling. However, on healthcare and medical surveys of patients about specific diseases or the quality of communications with doctors, etc. The respondents beiieve MR studies are very reliable and cooperation rates are much hgher than norm probably in the 60% range.

  5. I agree with Steve; perhaps credible is a more appropriate word. Either way, the problem is that the general public (including those who have been “educated”) have little understanding of what market research does and even less understanding of the statistical tools researchers employ to describe phenomena. To the public, statistics is about as credible as voodoo; thus causing a large amount of cynicism. Another phenomena to make my case: since the public doesn’t understand statistics and “good” research methods , many believe that research can literally prove anything; discrediting it and relegating research to a pseudoscience.

  6. Thank you all for your comments. I want to note that the omnibus questions were geared more toward the trustworthiness of various sources of market research and not toward any particular data output (see below). Maybe I should have titled the posting, “How Reliable Does the General Public Perceive Market Research Results to be When Attributed to Various Sources?” But that’s kind of a mouthful — and it wouldn’t fit on one line across the top of the page. 🙂

    The main point I was trying to make was that no matter how strong your data is, the source of that data makes a big difference in lay people’s perception of it. Thank God my work is only reviewed by my managers and clients and not by the general public!

    PMR1. How reliable do you perceive market research results to be when you hear them or read about them overall? Please rate on the scale below (7 is “Very Reliable”, 1 is “Very Unreliable”)

    PMR2. How reliable do you perceive market research results to be when they are attributed to the following sources? (respondents shown list of 9 sources with same 7-pt. scale)

  7. Only half think data from scientific journals is reliable? MR may be a respectable third, but being third when everyone looks pretty ugly isn’t so great. We all s*ck, but some s*ck even more.I agree with Nicholas that we probably get a negative halo from highly visible political polls. I would be curious to see a crosstab of these results by those who have A) participated in a political poll in the past 3 months vs those that have not and B) by education level.

  8. For your reference – I elected to show Top 2 Box results rather than Top 3 – although the order of the sources is exactly the same. Kathryn, to your point, even at Top 3 Box the ratings look pretty ugly.

    Top 2 Box | Top 3 Box
    Overall = 24% | 56%
    Scientific Journal = 47% | 70%
    University = 35% | 64%
    Market Research Company = 31% | 59%
    Non-Profit = 29% | 58%
    News Magazine = 23% | 52%
    Newspaper = 19% | 45%
    Popular Culture Magazine = 13% | 32%
    For Profit = 12% | 32%
    Political Organization = 8% | 20%

  9. Kathryn, good comments. I would like to see the data by your factors as well as by those who have ever participated in any survey versus those who have not. It would also be interesting to see reliability of government survey, commercial survey, survey about healthcare, finance, travel, etc. I would think these types of surveys would score higher as sources than just a general Market Research Company.

  10. Thanks Kathryn and Nicholas. Those would be interesting factors to run in a follow-up November Morpace omnibus survey which will field in a few weeks. I will keep you posted on the results if I am able to enter those questions in time.

  11. Thanks Greg. It would be great if you could address some of the items that Kathryn and me are interested in a future study.

  12. Hi Greg.

    Interesting article! Our Entrepreneur Club will soon be opening “The Test Market” as a permanent on-campus store. It’s a tool for consumer products manufacturers to reach the 18 to 24 yr old market with their newest products. We plan to sell, display, beta test, and promote their coolest products. We can instantly penetrate markets with free trials and promotional items to the 25,000+ ethnically and economically diverse So. Ca. students that come through our halls every semester. Other planned services: capture detailed demographics from registrar data, provide first reaction videos of students encountering new products, and recruit and select participants from over 100 academic programs for focus groups and beta tests according to the needs of our clients.

    A measurement of the reliability of our data seems like it would add significant credibility to our presentation to prospective customers. But, it sounds like reliability is a common industry problem. We’d thought about asking to get one of our interested cosmetics clients to test little known products that have already proven themselves to be successful, along with unsuccessful lines, just to see if we would have predicted each.

    Sounds effective, but expensive, with little other benefit. Any ideas? Would you be willing to point our students to any prior studies or approaches? Or, do you think we’re worrying about something we don’t need to worry about?

    Our students don’t pretend to have any test marketing knowledge, but follow directions well, and learn quickly. Do you think our prospective clients will be turned off when we tell them the project design, and reliability of results, rests on them?

    Jim Sugden, CPA
    Professor of Accounting – Orange Coast College
    Faculty Advisor – Business and Entrepreneur Club

Join the conversation


Greg Deinzer

Research Director, Morpace