Editors Note: We usually shy away from publishing content that has been featured elsewhere before (although we do like to share content with partner sites), but this post from Andrew Jeavons is so well done and pertinent to the themes we explore here at the GreenBook Blog that we felt it was worth an exception. We hope you enjoy it. LFM
By Andrew Jeavons
I enjoyed Jerome Sopocko’s article, he is a very intelligent and interesting person. He makes an interesting case for protecting the MR business. I don’t agree with it though. If any industry has to resort to protectionism then it deserves to die.
There is a lot of noise about ‘DIY research’, most of it seems to be about how it is a ‘threat’ or is somehow wrong. Neither of these is true, unless you want the world to stay the way it is. Progress is always going on, and to decry something for which there is a need is simply not going to work. MR has to embrace change. You can’t stop the future.
I started off as a psychologist, and then I moved into neuropsychology, statistics and eventually software development for the MR industry. I agree that mathematical models are great things. They can have great elegance and power. The fact is that the example that Jerome gives of modeling is, generally speaking, the exception for MR. The vast majority of MR studies contain nothing so elegant or durable. So let’s be clear what we are talking about. While there is some tremendous research and modeling going on in MR, this is not the norm.
I can’t really take the argument that ‘writing surveys’ is a vital justification for ‘professional’ MR. If that is really one of the core unique selling points for ‘professional’ MR, give up now – seriously. Yes, you can bias results via wording. However the assumption than any survey can access some absolute truth is invalid, surveys are very general blunt instruments. The whole idea that we can accurately introspect as to why we do something via a survey is naive psychology of the worst kind. It’s not a question of dumbing down surveys, it is a question of being realistic as to what sort of data you get from a 45 minute survey with huge grids in it. People will always give you answers, but they may not be relevant answers.
The reason why DIY MR is important is that there is a demand for it. MR has adopted a very condescending attitude to DIY research. Just because companies want to do research themselves without the gods of MR blessing their efforts doesn’t mean they are doing the wrong thing. It all comes down to money. There are tens of thousands of companies that don’t have tens of thousands of dollars to give to MR companies for research. Does this mean they can’t research their market place? Just because I can’t afford to buy a BMW doesn’t mean I shouldn’t have a car. It is absurd paternalism to say that companies can’t do their own research. I reject the argument that DIY research is inherently a bad thing. There may be bad DIY research, just as there is bad professional research.
What traditional MR companies have to accept is that making money on the mark up of data collection costs is an old business model. The traditional MR business model generally had a lot of profit and revenue associated with data collection with a relatively small amount of the revenue associated with analytics or modeling. It is clear that MR can’t use this model anymore, or at least it can’t use it all the while. Analytics is the new frontier, whatever we think of the quality of data, we can get the data. What to do with it is the next question. I would suggest that in the long term MR professionals should forget about criticizing survey design and concentrate more on honing their analytical skills.
MR has to adapt, and the DIY movement represents an ideal opportunity for MR companies to take on a consultative role. It isn’t how things used to be, but nothing is the way it used to be. I don’t see a movement by statisticians to disallow the sale of end user statistical software. I don’t hear anything about ‘DIY statistics’ being a bad thing, and that is what most of the people in MR do when it comes to statistics. Professional MR practitioners are often guilty of grievous abuses of statistics because they, either through lack of training or experience in that area, are ‘DIY’ statistics users. How can they therefore complain about ‘DIY’ research?
An example of the DIY nature of the so called MR professionals is the practice performing multiple t-tests on a cross tabulation without adjusting the significance level for each of the t-tests performed. To statisticians this is one of the problems of ‘DIY’ statistics. Yet this is a common practice in MR. MR professionals seldom pay any attention to the assumptions of parametric statistics. I can’t recall when I saw any analytics using non-parametric statistics, yet a great deal of data in MR simply does not conform to the assumptions required by parametric tests. When was the last time you saw someone report a Wilcoxon test result? A Mann Whitney U test anyone?
This is just an example to illustrate that a lot of current MR is DIY. All sorts of disciplines such as statistics, anthropology, psychology and sociology (to name but a few) are plundered at will by so called MR professionals. Various bits of these disciplines are then regurgitated to their clients, often distorted and misunderstood, in the hope of ‘telling a good story’. There is more Scheherazade than science in MR it seems. I therefore find it hard to take the arguments of protectionism from professional ‘MR’ concerning ‘DIY Research’ when many of the ‘professional MR’ skills are obvously DIY.
MR has to embrace reality, and that includes an increase of so called ‘DIY’ research. The paternalism and condescending attitude just looks like a defensive reflex. Maybe all this means is an adjustment to the way MR works with clients, is this such as bad thing? What is certain is that decrying something that is happening already, and for which there is a steadily increasing demand, is a sure way to become irrelevant.