Research Methodologies

June 23, 2011

Has Gongos Research Proven The Validty of Smartphone Research Or Not?

Dr. Michael Alloto and Ray Poynter debate the results of a new study from Gongos Research on the validity of smartphone research.

Leonard Murphy

by Leonard Murphy

Chief Advisor for Insights and Development at Greenbook

0

 

UPDATE 06/29: Due to the overwhelming interest in this topic we’re working on putting together a virtual debate featuring Dr. Alioto from Gongos Research, Ray Poynter of The Future Place, Reg Baker of Market Strategies (who posted on this topic last week) and myself on July 7th at 12:00 PM EST. Read all about it & register to attend here. See you there!

 

Yesterday Ray Poynter posted a response to the press release by Gongos Research regarding the results of a new study they conducted on the validity of smartphone research. You can find the original press release here.

Ray questioned the claim in his post in his usual highly erudite way:

OK, somebody had to blink! For the last few days there have been several tweets and comments using the phrase “Smartphone surveys prove their validity in marketing research”. When I saw the first post I immediately classed it as harmless hyperbole. But when I saw that MSNBC had used the same headline (14 June 2011) I felt I had to shout out “The Emperor has no clothes”.

The stories seem to track back to a press release by Gongos Research, released on 14 June 2011. The press release includes the phrase “a new study proves that smartphone-based survey data is statistically comparable to online survey data.”

This claim in linguistically and methodologically ludicrous, a single study cannot prove that something happens, it can show that it sometimes happens, but that does not prove a positive. A single, well designed, study can show that something is NOT true. For example, the claim that all swans are white is disproved by finding one black swan, but finding one more white swan does not prove all swans are white.

If we assume, for the moment, that the press release means that Gongos Research conducted a study, looking at a wide range of questions types, in one or more markets, with one or more types of customers, and that the study showed the results were acceptably similar, then we would not have proof of the validity of smartphone research. We would have evidence that smartphone surveys can sometimes work. If more tests are conducted (and ideally other agencies should be willing to pay for them) we can start to find out if smartphone surveys are ‘often’ (or better still ‘usually’) acceptably similar to other modalities. If we had the results of say 20 studies and they all showed that smartphone research was acceptably similar to other modalities we would not have proved it, but we would feel reasonably confident about using it, even 10 out of 10 positive results would make us pretty happy about trying it for a live study.

I do hope that Gongos Research publish their research as it is potentially really helpful to the industry, as evidence about the viability of smartphone research.

 

Dr. Michael Alioto, the principal author of the report sent me his response and posted it on Ray’s blog. Here it is:

 

I appreciate your comments, Ray. Indeed, our intent was to shed light on the currently existing body of evidence in our industry – not claim a “proof” in the Philosophy of Science or mathematical sense.

Much of the literature surrounding the use of mobile or Smartphone devices for survey research is mixed. This study indicated that data can be collected and compiled in multi-mode formats with confidence, which enabled us to support some of the accepted approaches and question others.

But let’s not lose sight of the real issue. While we have solid evidence to support the use of multi-modal platforms for research, the question of migration from online to smartphone as a preferred platform is still to be determined.  Many of us in the marketing research discipline agree that online surveying has its issues (biases associated with representation, scale uses, artificial interviewing environment, etc.).  These issues have become increasingly apparent as the online platform has been less able to solve “puzzles” in the Kuhnian sense of the term.  The less puzzles solved, the more we are inclined to view the smartphone platform as a potential replacement for online surveying (assuming that the smartphone platform will be able to solve more of the puzzles identified by the online methodology and other potential platforms, such as social media, etc. are not direct competitors for the next platform paradigm).

The importance of this study for the marketing research industry is that we were able to rigorously test a number of assumptions and items between the two separate platforms.  But this is not the end.  Similar to your thoughts, we welcome other research agencies to join in rigorously testing data and research elements between the two platforms.  Based on our findings, we believe that these additional tests will support the use of smartphones as a valid survey platform.

The critical nature of the future of marketing research surveying is at hand. With technology progressing forward and response rates continuously falling, we need to be able to gather valid and reliable information from our respondents. We stand behind smartphones being a valid avenue to accomplish this goal.

We plan on rolling out the key findings from this study at the MRMW11 conference. Ray, we would love to have you join us and contribute to the debate and good discussion.

So, there is the current state of what is happening leading up to a much needed and critical debate. I’d like to suggest to both parties that perhaps we continue this at the Market Research in the Mobile World Conference next month, or if that is not possible perhaps stage a virtual debate around the same time?  It’s time to tackle these issues head on; the future of the market research industry is too dependent on mobile and social media-based technologies  for us not to exert the needed time and energy to answering these questions and moving on.

 

0

innovationmobile researchmobile surveysmobile research

Disclaimer

The views, opinions, data, and methodologies expressed above are those of the contributor(s) and do not necessarily reflect or represent the official policies, positions, or beliefs of Greenbook.

Comments

Comments are moderated to ensure respect towards the author and to prevent spam or self-promotion. Your comment may be edited, rejected, or approved based on these criteria. By commenting, you accept these terms and take responsibility for your contributions.

More from Leonard Murphy

CEO Series: AI Integration and the Future of Marketing Insights with Alex Hunt, CEO of Behaviorally

CEO Series

CEO Series: AI Integration and the Future of Marketing Insights with Alex Hunt, CEO of Behaviorally

Explore the power of AI in marketing with behaviorally's CEO, Alex Hunt. Learn how to leverage predi...

The Next Wave of Disruptive Technology that Changes Everything

Research Technology (ResTech)

The Next Wave of Disruptive Technology that Changes Everything

There have been a few big inflection points of societal disruption driven by technology in the last 50 years: One was the introduction of the Internet...

Leonard Murphy

Leonard Murphy

Chief Advisor for Insights and Development at Greenbook

Quantifying the Impact of Insight Innovation

Insights Industry News

Quantifying the Impact of Insight Innovation

We previously announced the milestone of our Insight Innovation Exchange (IIEX) conference series’ 10th anniversary, celebrating a decade of identifyi...

Leonard Murphy

Leonard Murphy

Chief Advisor for Insights and Development at Greenbook

How Generative AI is Changing the Research Industry

The Prompt

How Generative AI is Changing the Research Industry

ChatGPT (GPT meaning “Generative Pre-trained Transformer”) is a chatbot launched by OpenAI in November 2022. It has quickly exploded in public awarene...

Leonard Murphy

Leonard Murphy

Chief Advisor for Insights and Development at Greenbook

ARTICLES

Moving Away from a Narcissistic Market Research Model

Research Methodologies

Moving Away from a Narcissistic Market Research Model

Why are we still measuring brand loyalty? It isn’t something that naturally comes up with consumers, who rarely think about brand first, if at all. Ma...

Devora Rogers

Devora Rogers

Chief Strategy Officer at Alter Agents

The Stepping Stones of Innovation: Navigating Failure and Empathy with Carol Fitzgerald
Natalie Pusch

Natalie Pusch

Senior Content Producer at Greenbook

Sign Up for
Updates

Get what matters, straight to your inbox.
Curated by top Insight Market experts.

67k+ subscribers

Weekly Newsletter

Greenbook Podcast

Webinars

Event Updates

I agree to receive emails with insights-related content from Greenbook. I understand that I can manage my email preferences or unsubscribe at any time and that Greenbook protects my privacy under the General Data Protection Regulation.*