Download the

A Bayesian Approach to Modeling the Market Research Conference Selection Dynamic


Editor’s Note: OK, before you dive into this post there is a back story here. Originally this was slated for publication in a column for Marketing Research magazine (an AMA publication) titled “Beg To Differ.” The AMA rejected it and the author and associated players in the resulting debacle approached us to ask if we would publish it in the interests of getting this viewpoint out into the market. I agreed because I believe GreenBook Blog is (and always will b)e a platform for open, transparent debate on all things that impact our industry and I happen to agree that we need to let our conferences evolve to meet the needs of a changing marketplace.

There is a fair amount of commentary associated with this piece that I have posted in the comments section below the article; they are well worth a read for their erudition on the core issue and to get a picture of the chain of events that led to a pretty significant shakeup within one of our major publications and trade orgs.

I want to be clear here: by no means am I or GreenBook taking any sides here other than supporting the desire for transparency and truth. There are two sides to every story and I hope by publishing this we can act as a catalyst to get an open dialogue going on how all stakeholders in our industry can work together through the process of dialectics to move us forward and make us better than we are today.

Besides, this is a pretty entertaining post Dick has written and it hits the nail on the head in many ways. Enjoy!


By Paul Richard “Dick” McCullough

Do you find this post’s title impressive? Ostentatious?  Incomprehensible?  Your answers (multiple responses allowed) will indicate which conferences you should, and should not, attend.  I’ll explain further down.

Marketing research conferences can be divided into two broad classes: Vacuous and Substantive. Please note I’ve excluded invitation-only conferences from this taxonomy.  I’ve also excluded seminars and workshops.

Almost all marketing research conferences in North America fall into the Vacuous class.  This includes every conference sponsored by companies whose primary business activity is sponsoring conferences and most conferences sponsored by anybody else.

Why are Vacuous conferences so popular?  Same reason as focus groups:  They’re easy to understand and there’s always free food.  The purpose of these conferences is not education, it is networking.  Vacuous conferences are giant, costly cocktail parties.  Vendors glide through the crowd like sharks smelling for injured fish, seeking any opportunity to pass out a business card.  Corporate researchers, especially young ones, wander in wide-eyed and excited, hoping to learn something and/or meet someone who might help their careers.  The speakers, almost always corporate clients (vendors willing to pony up expensive exhibitor fees sometimes sneak onto the agenda, too), prepare entertaining, colorful presentations that showcase their marketing acumen with the intent of dressing up their resumes and perhaps meeting someone who might eventually help their careers.  Of course, standing up in front of a large group of your inferiors and pontificating isn’t bad for the ego, either.

And for everyone, a three and a half star hotel, some nice weather in February, several dinners at excellent restaurants, lots of cocktails and a few days out of the office, all on the company tab.  And don’t forget the frequent flyer miles.  Could be worse.

The brochures do try to give the impression that you’re going to learn something but rarely is anyone fooled.  Those that are fooled, like I was early in my career, quickly realize these are not my kind of conferences, or perhaps more precisely, that I’m not their kind of attendee, and move on to the other conference genre: Substantive.

The class of Substantive conferences in North America includes two that I would like to contrast here: the Sawtooth Software Conference, held every 18 months and the AMA’s Advanced Research Techniques Forum, aka, the ART Forum, held annually.

Sawtooth Software, as their name implies, is a software company.  They author the leading conjoint software in marketing research, as well as other products.  Although it would be in their short-term best interest to showcase their products at this conference, the opposite is actually true.  The conference gives highest priority to papers using non-Sawtooth techniques.  The intent of the conference is to educate.  Each presentation must be designed so that there are some useful takeaways for each person attending the conference, regardless of experience and education.  That means the presentation has to be basic enough for beginners and advanced enough for award-winning PhDs.  Each presenter is required to provide a journal quality paper as well as the presentation deck.  Each accepted submission is assigned a supervisor who reviews the presentation in draft form and makes suggestions to insure both the final presentation and paper are appropriate for the audience.  It is a major effort and not all presentations/papers are successful.  But most are.  The intent of the conference is to present original research that is relevant to both the practitioner as well as academic communities.  The conference occurs every 18 months rather than annually because, among other reasons, Sawtooth has found that from Call-for-Papers to publishing conference proceedings is well over a year cycle.  Good research sometimes takes longer.

The ART Forum is, on the surface, similar to Sawtooth.  According to the AMA website, “Experienced research practitioners who use advanced methods in their jobs comprise the largest segment of [ART Forum] conference attendees.”  I believe that the ART forum was originally designed to disseminate academic ideas to practitioners, and provide academics with practical problems for which they could develop better solutions than currently practiced.  It was a forum for two-way communication between the two communities.  Both sides benefitted.

I remember attending ART Forums where I came home with a head swirling with new ideas, excited to investigate some new technique or variation. Recent ART Forums, at least for me, have been filled with inscrutable presentations that wouldn’t be relevant to my practice even if I understood them.  The fact that the ART Forum does not publish papers to support the presentations makes the presentations even more difficult to comprehend.   There is no opportunity to review carefully, slowly and repeatedly the complex analysis presented verbally.

A typical ART Forum presentation nowadays, seems to follow this script: 1) pick a very specific consumer behavior, eg, buying high-end fleur de sel from online gourmet retailers using a smartphone, 2) hypothesize a mathematical model to describe that behavior (note: it is critical that this model differ from any other model ever specified), 3) display a screen full of Greek letters that allegedly define the model, 4) state during the presentation that the model is extremely straightforward and quite easy to estimate, 5) take several months or years to estimate the model, 6) present findings that show this custom-coded model performs better than current alternatives and 7) select a paper title that rivals that of this column for pomposity.

Exactly who would find this relevant, useful or even interesting?  To be honest, I know there are some brilliant minds on both the academic and practitioner sides that do find these presentations interesting.  Maybe even relevant, although that’s harder for me to imagine.  But I also know that’s a very small group.  And I know I’m not in that group.

It seems to me that someone needs to admit that the ART Forum has gotten too academic.  I’m not smart enough for today’s ART Forum.  And I think enough of myself to believe that if I’m not smart enough, there are a lot of other researchers who aren’t smart enough, either.  I think the ART Forum is playing to an ever-dwindling audience.  It is in danger of becoming irrelevant, at least to practitioners.

Now back to my title.  If you understood what it meant and thought it was impressive, you should attend the ART Forum.   You might learn something you find interesting.  If you didn’t understand it but you thought it was impressive, you should still attend the ART Forum.  You won’t learn anything at all but you can brag to all your friends and clients that you attended.  If you understood this column’s title but thought it was laughingly pretentious, then the Sawtooth Software Conference is your best bet.  You’ll return home a better researcher than you left.  And finally, if you simply had no idea what any of those words meant, just about any of the Vacuous class conferences will do (remember to wear your name tag at all times).

Finally, for the careful readers who realize something promised by the title is still lacking, let me add this: review your answers to the question in the first line of the column.  If you would answer differently now, change your answer, reread the column and repeat until the answers converge.  A simple man’s Bayesian model.

Please share...

13 responses to “A Bayesian Approach to Modeling the Market Research Conference Selection Dynamic

  1. Response to “A Taxonomy and Assessment …” by Dick McCullough

    By Greg Allenby

    Dick McCullough makes some great points. I, like him, have been a regular ART Forum attendee for 20 years, and have also attended a number of Sawtooth Conferences. We both come away from these conferences with new ideas, our careers have benefited from the people we’ve met, and we’ve gained much from the opportunities given to us to present our work. We count many people who attend the ART Forum as our friends, and we look forward to seeing them each year. The ART Forum is my favorite conference.

    I think what Dick senses can be chalked up to the natural course of events. Each of us is trained with a set of skills that are popular at the time we went to school, and, as time goes by, new techniques, perspectives, data and problems come onto the scene. Every seven or eight years (the time needed to get tenure) brings with it a new generation of thinking and problems. I, along with many others, helped to popularize Bayesian methods in marketing in the 1990’s, and the field of marketing is now on to new methods and new problems. Things that are popular now are social networks and structural economic models, and these problems are not necessarily tackled with the help of Reverend Bayes. As the latest shift occurred, I noticed myself grousing that those youngsters just weren’t working on interesting problems – i.e., I mean, who would actually use that stuff? I was beginning to sound like Dick.

    Each year, the AMA Research Council appoints a chairperson to oversee the ART Forum conference. And, each year, the chair assembles a committee of academics and practitioners to develop the conference program. The chair is always looking to add new people to the committee to welcome and include the new generation of thinkers and doers, and the charge is given to assemble a program that reflects three things – rigor, relevance and emerging topics.

    Academics love rigor. Developing variables and models that are good for understanding consumer behaviors is what drives us. We want to think deeply about how variables come together to prompt purchase, how decision options influence choices, and how firms might leverage this knowledge to their advantage. We love to drill deep wells.

    Practitioners love relevance. They want to be useful in today’s world for many reasons, including the ability to get paid. They have a keen sense for important problems and have the greatest understanding of why academic theories don’t work. Because practitioners work at the point of application, they have the most to contribute to the development of new theories. Practitioners are the ones who tell us academics the well is dry and its time to move on, much as Dick is doing in article. This is one of the benefits I get from the ART Forum – I learn what makes practitioners frustrated with us.

    Everyone loves emerging topics. We all like to think about new data, new models, new methods of estimation, and new approaches to exploring implications of our analysis. I know that many of the emerging ideas presented at the ART Forum won’t work out, and I gain much from confronting myself with ideas that bring me to new fields of exploration.

    Balancing the mix of rigor and relevance in both old and up-and-coming topics is a difficult task for the ART Forum program committee. It is especially difficult when balanced against the need to develop the next generation of academics and practitioner who will be the future leaders of the field. Looking back on my early years with the ART Forum, I remember legendary characters such as Bill Neal, Rich Johnson and Jordan Louviere guiding the content and the discussion at the conference. These individuals had a perspective, wanted to articulate it and were outspoken about what was relevant. They were also very welcoming and encouraging to me, helping me to develop as an academic interested in applied problems.

    I agree with Dick that the ART Forum needs to elevate the banner of relevance while continuing to hold onto rigor and emerging topics. This is a difficult task, one that requires a sustained effort over an extended period of time. My suggestion to the AMA is to consider a three-year program committee made up of practitioners and academics with track records reflecting rigor and relevance, plus an appreciation of emerging topics. There needs to be balance among these factors, along with the desire to develop the next generation of leaders. I believe that the movement to a multi-year committee would better allow for the balance needed to satisfy all attendees while ultimately leading to new marketing thought and practice.

    BY William D. Neal
    I certainly enjoyed reading Dick McCullough’s editorial and I am in total agreement with almost all of his points. Been there, done that, over and over and over again.
    I do think there are a few exceptions to his taxonomy, most notably the annual MRA conference, where there is a very healthy blend of activities of both the selling and networking type and of the substantive learning type. But in general, he is right on the money.
    Initially, I planned to respond to Dick’s article with a dissertation on the growth, progression, and death of research conferences in North America and their impact on the marketing research community. But that quickly got too long and far too boring. So I will just cut to the chase – what went wrong with the ART Forum?
    In a nut shell, as Dick so bluntly stated – it has lost its relevance – at least its relevance to practicing methodologists. Before going on, I need to disclose that I have not attended an ART Forum for many years. But, I do study the conference brochure in detail each year and weigh the potential gain with the potential pain. The pain has been winning out for many years now. However, this last conference showed a lot more potential gain than previous ones, and I would have attended if not for a prior engagement.
    The original ART Forum had a dual mission:
    1. Bring together practicing methodologists and methodologically inclined academics to address common problems and common solutions to real world marketing issues.
    2. Act as a conduit for knowledge transfer from theory to application, and in the process provide feedback on what worked and what didn’t work in the real world of marketing.
    The original ART Forum was based on a concept extension of Rich Johnson’s Sawtooth Software Conference. And, I need to make sure the record is straight – Rich Johnson is also the “father” of the original ART Forum. It was his concept and his execution that we were attempting to duplicate to extend the coverage of subject matter beyond that which was traditionally being covered at the Sawtooth Conference at that time. It worked, for a while. All of us who developed the initial design of the ART Forum were dedicated attendees and participants in the Sawtooth Software Conference, and we were deeply appreciative of all the work and care that went into its planning and organization. Our motive was to supplement Sawtooth, not replace it.
    Those original ART Forums were characterized by impassioned debates on methodological issues, irreverent intolerance for any “black box” presentations, and a demand for methodological transparency. You needed a tough skin and a quick mind to be a presenter, lest you be eaten alive by the audience.
    By design, there were long breaks between sessions that allowed for continuing open-ended discussions after each presentation and informal discussions outside the presentation hall. And we all had a great time and learned immensely in the process. We also had the opportunity to get to better know our fellow methodologists – both practitioners and academics.
    The academics that attended were almost exclusively well-published, tenured, dedicated teachers who kept their feet firmly planted in both the academic and the practitioner worlds – folks like Paul Green, Bill Dillon, Jordon Louviere, Joel Huber, Vijay Mahajan, Greg Allenby, and many others too numerous to mention. Their questions and observations were invaluable to the learning experience.
    So what did go wrong? More importantly, why?
    Has it gotten too academic as Dick suggested? Yes, I believe so. Having five presenters listed for one 30 minute presentation and a 10 minute Q&A is a sure sign that a bunch of people are “resume building.” But being “too academic” is an insufficient descriptor. I’ve been to a lot of academic conferences where I’ve learned a lot. I think the problem is more one of relevance. The conference has lost its focus on practical issues and applications – those that have relevance to the real world of marketing. The models and modeling techniques have become more important than the outcomes and their applicability to practical issues. (Again, the 2012 conference appeared to be an exception, based on the presentation titles and presenters.)
    As Dick also implied, there is the generalizability issue – can the methods and procedures presented be generalized to a class of problems that we know already exist or are likely to exist in the environment? It seems to me that the generalizability issue has taken a back seat to the perceived elegance of the model or technique. Seldom do you see the presenter who has walked their pet technique through multiple scenarios and multiple samples.
    In the early ART Forums we put a premium on finding presentations where a generally accepted model or technique didn’t work. We assumed that failures were of high educational value and provided an opportunity to reinforce the skepticism that is a necessary trait for our field. I fondly recall Steve Cohen standing in front of the audience stating that he “tortured the data until it confessed.”
    But all this has to do with the presentations themselves, which is a relatively easy fix.
    In my opinion, the real underlying problem is the management of the conference. This is where ART Forum has diverged significantly from the Sawtooth Conference. The Sawtooth conference was conceived and actively managed by Rich Johnson for many years. Subsequently, the Sawtooth staff remained true to Rich’s vision and management style.
    In contrast, the ART Forum is managed by an ever-changing committee selected by an ever-changing Marketing Research Division Council and the AMA staff. The original vision has vanished and the continuity of focus seems to have vanished with it. Maybe David Bakken and David Lyon will respond to this observation since they have been on recent ART Forum committees and are long time attendees.
    There seems to also be a scarcity of interaction time. There is no cocktail reception, the breaks between sessions have vanished, and there seems to be little time available for the often workplace-isolated methodologists to renew friendships and make new ones. I believe this was one of the most important features of the ART Forum and contributed immensely to the learning experience. But it seems to have disappeared. Even the unsanctioned floating poker game of those early years contributed greatly to building friendships and lasting experiences.
    ART Forum seems to have lost its grove. Or maybe it’s just that Dick and I are getting older and more curmudgeon-like. Dick?

  3. By Paul Richard “Dick” McCullough

    Until recently, I wrote a column for Marketing Research magazine (an AMA publication) titled “Beg To Differ.” The column was intended from its inception 3 years ago to take a contrarian view in order to raise awareness and encourage debate on issues relevant to the marketing research community. Marketing Research encourages readers to respond to any articles in its BackTalk section.

    After attending the latest ART Forum (June 2012) and listening to numerous distinguished marketing science practitioners voice the same frustrations that I was feeling, I wrote a column critical of the ART Forum in its current form. I vetted the column with five colleagues with unimpeachable credentials prior to submission. I also arranged for Bill Neal and Greg Allenby, two well-known, highly decorated members of the marketing research community to write responses. Bill and Greg are both past winners of the Parlin Award, the highest honor the industry bestows.

    Chuck Chakrapani has been the long-standing editor-in-chief of Marketing Research magazine and is generally credited with building the magazine into a highly respected publication.

    AMA staff unilaterally decided to not publish the ART Forum column or the accompanying responses. Mary Flory, managing editor of AMA magazines, informed Chuck of the decision in a brief email. There was no discussion with Chuck about this decision and there was no rationale given.

    Chuck considered the AMA staff decision to not publish a column critical of an AMA event as censorship and, consequently, he resigned the following Monday, September 24. I resigned the same day. Ken Deal, author of the Marketing Research column Software Review, resigned two days later. Dave Lyon, member of the Marketing Research Editorial Review Board resigned the following Monday.

    Bill Neal has led a vigorous effort to convince the AMA that they had made an error in judgment. Recently (October 4), Dennis Dunlap, CEO of the AMA, and David Reibstein, chairman of the AMA, jointly issued an email apology claiming the AMA action wasn’t censorship and expressed willingness to publish the column and responses in Marketing Research magazine. In an email co-authored by Bill Neal and me, we suggested several steps to be taken by the AMA to rectify the situation and ensure censorship could not happen in the future. A response from the AMA is expected shortly. In a separate email, I informed the Mary Flory that I have not given the AMA permission to publish the column.

    Perhaps this is the end of the story. Perhaps not. This “diary” will be updated as further events dictate.

    Please take the time to read the offending column and responses. Then share your opinion, whatever it may be, with Mary Flory, managing editor of AMA magazines ([email protected]).

  4. I think negative feedback is important. We can’t always be all things to all people, but at the MRIA if you think anything sucks or has lost relevance. We want to hear about.

  5. As to the controversy, it is a bit confusing. Having read the article, I could believe the editorial staff rejected it for reasons other than censorship (there are lots of reasons why journals may reject or request rewrites of articles) . But given that an apology was later issued, the issue becomes muddy.

  6. Kathryn,

    I am curious. Other than not wanting a commentary critical of the AMA’s ART Forum being published in MR, what reasons for rejection that are NOT censorship are you referring to? Why was no reason given? Why did they overrule the longtime editor? Why wasn’t it discussed? The commentary was part of an agreed upon contrarian point of view regarding research… What do you think was in their minds. I’m curious.

  7. The controversy is simple. AMA staff unilaterally rejected a submission that the editor-in-chief felt was appropriate. There was no discussion with the editor and there was no explanation given. There was no request for rewrites, either. The apology included a specious explanation that MR was not an appropriate venue for a discussion of the ART Forum. I know of no marketing scientists who agree with that POV and I know several Parlin award winners who disagree. I believe the central issue is who should decide magazine content, the editor or AMA staff?

    1. I have to chime in here to say that although I have published several pieces that my colleagues at GreenBook were not thrilled with for a variety of reasons, they have always supported my editorial integrity. Hopefully this is all some type of misunderstanding, but it certainly sounds like this wasn’t handled well and created a much bigger issue than the original critique was.

  8. @Thomas, I could answer your question about why I can see legitimate reasons this article could have been rejected–but of course I can not answer questions about the surrounding events (I wasn’t here and there are always two sides to these stories). Even so, I learned long ago, never poke an angry bear. And I sense a lot of angry bears lurking 😉

  9. The problem, Dick, continuing on your taxonomy, is that there are two types of people in our business – those that think we’re doing something resembling science and those who don’t. The latter by far outnumber the former these days, I fear, which may be part of the decline and fall of the MR conference.

  10. Hi Paul, (and Steve) I think the assumption of two types is a little presumptuous, unless you mean conferences you approve of and conferences you don’t. There are events that focus on meaty contributions (and I too hold Sawtooth in high regard here), there are events that focus on new ideas, there are those that focus on networking, there are those that showcase what difference parts of the industry are doing, and there are those that facilitate group working and discovery (there are also some dire ones where people are allowed to try and sell their product). There are also events, like the recent ESOMAR conferences in Atlanta and Mexico that combine several elements, with the serious material being covered in the workshops, useful panel discussions, cross-cultural reviews, and good networking.

    One of the key drivers of conference formats these days is money. There is a large amount of competition and many companies seem to have ever smaller budgets for conference attending. This results in sponsorship being a key source of revenue, which means the need of sponsors is the biggest driver in the form of the conference. In many cases the sponsors are software and panel companies, so they want lots of researchers to attend. The best way to get lots of researchers to attend is to have lots of end-clients attending, and the best way of getting end-clients to attend is to have them as speakers, which drives the nature of the papers and presentations in a particular direction.

    The final problem for the sort of event that you enjoy is that methodology heavy conferences tend to attract far fewer attendees, which makes them less attractive to anybody seeking to make money from organizing them. I suspect that one of the reasons that Sawtooth has been able to stay true to its mission is because it is not a democracy or a trade body, it is a private company – i.e. it does not have to please the wider membership and can put on the sorts of events it wants to – subject to getting enough people to attend.

  11. I’ll resist diving into the controversy here, but I wanted to add that in the category of the most substantive of conferences, it would be remiss not to include the Qualitative Research Consultants Association (QRCA). The conference is able to avoid everything that Dick describes as vacuous, largely by only admitting research practitioners and not buyers. It’s a tremendous educational opportunity for qualitative researchers.

  12. The Advanced Research Techniques Forum (ART Forum) was never intended to be a typical conference. Rather, those of us who created it saw it as a true “forum” – a meeting of like-minded professionals to discuss and debate common issues. It was designed to be a true two-way exchange of ideas among practicing research methodologists, which is a small, but highly influential group, within our profession.

    Traditionally, ART Forum has been a very big success for AMA. It generated good net income, and more importantly satisfied the needs of an important constituency within AMA membership.

    Dick McCullough’s original article was critical of how the ART Forum had lost its original vision and had compromised its mission. I agreed, and I know many others who also agreed. Dick’s contrarian section in Marketing Research magazine was the ideal forum for raising that issue. Had it been published, I am quite certain that many others would have chimed in and offered observations and recommendations on what should be done about it. It should have been a valuable exchange among professionals on addressing the problems and finding workable solutions.

    What happened is that the AMA staff took great umbrage that one of their products had come under criticism, and unilaterally overrode MR’s long-term and highly respected editor, Chuck Chakrapani. That was blatant censorship – no discussion, no compromise – just cut it.

    Chuck promptly resigned the editorship of the magazine, as I believe any other editor of an AMA publication would have done under similar circumstances.

    In a letter that Dick and I penned to AMA’s CEO and Chairman, we recommended a set of steps we believed AMA should take to address these issues. There has been no response to date of which I am aware. Now the cat is totally out of the bag, and there is no way to put it back. The AMA staff has received a big much bigger black eye then they ever expected – the unintended consequences of a stupid decision.

Join the conversation

Paul Richard McCullough

Paul Richard McCullough

Founder & President, MACRO Consulting Inc

Sign up to our newsletter

Don't miss out...
Get great insights content delivered straight to your inbox.
I agree to receive emails with insights-related content from GreenBook.

You can manage your email preferences or unsubscribe at any time. GreenBook protects your privacy under the General Data Protection Regulation.