Research Methodologies

January 3, 2017

What are Monte Carlo Studies?

Monte Carlo studies have made R&D for statistics and machine learning much easier and faster.

Kevin Gray

by Kevin Gray

President at Cannon Gray

0

By Kevin Gray

We owe London to Rome and, in a way, Statistics to Monaco. Quite a few of our earliest statisticians were mathematicians with a interest in games of chance and, in jest, I have even described myself as a professional gambler. (Attention wanders quickly, though, when I begin to describe what I really do…)

Some of you may have heard of MCMC – Markov Chain Monte Carlo – which is used extensively in Bayesian Statistics. There is actually a thick volume entirely devoted to that subject entitled Handbook of Markov Chain Monte Carlo (Brooks et al.). I won’t get into Bayesian methods in this post but Professor Andrew Gelman gives a nice overview on his blog in Bayesian statistics: What’s it all about?

So what are Monte Carlo studies? Though we may sometimes feel we’re drowning in data, truth be told, we frequently don’t have the data we need for specific purposes. This holds for university professors as well. Collecting and assembling new data takes time and money and the new data are not guaranteed to suit our objectives.

Say a statistician would like to evaluate a new (or existing) statistical procedure, machine learning algorithm or fit index. A single sample of real data will rarely be enough. Moreover, it would only be a sample and by definition subject to sampling error. What if, instead, we could create our own population and “sample” from it? Since we have designed the population ourselves, we know the answer in advance.

With today’s computers it’s easy to simulate thousands or even hundreds of thousands of samples of based on hypothetical populations (which are often inspired by real data). Importantly, it’s possible tweak these imaginary populations to see how well the new algorithm (or whatever) holds up under various experimental combinations of sample size, skewness, kurtosis, collinearity and what have you. Knowing the truth – because we’ve created it – allows us to accurately assess the algorithm’s performance and also compare it against alternatives.

Academic publications such as the Journal of the American Statistical Association and Structural Equation Modeling describe the results of Monte Carlo studies in every issue. One of my key takeaways is that many “classical” statistical methods are quite robust to violations of their assumptions and are nowhere near as fragile as often feared or claimed. Some do fall down badly, however, when they’d been expected to perform well. This holds for newer methods too. Applied statisticians like me need to keep our eyes peeled for new guidelines from the academic community. Not as exactly like watching MythBusters, but part of the job. We also occasionally perform Monte Carlo simulations to estimate the sample size required for statistical modeling we’re contemplating.

Monte Carlo simulations are also conducted by some political pollsters and analysts. A few of you might also associate the term with “Wall Street Quants.” I have a book on this latter, financial, application called Handbook in Monte Carlo Simulation (Brandimarte), and many others about this topic have been published in addition to hundreds of papers. Here, the basic idea is to generate a large number of forecasts based on an econometric model (or ensemble of models) under varying economic conditions or events that could plausibly occur.

Monte Carlo studies have made R&D for statistics and machine learning much easier and faster. Real data are still needed, but no longer do we have to rely on mathematical theory and wait years for empirical studies to tell us how a new technique stacks up against established methods or other new kids on the block.

I hope you’ve found this interesting and helpful!

0

machine learningsample

Disclaimer

The views, opinions, data, and methodologies expressed above are those of the contributor(s) and do not necessarily reflect or represent the official policies, positions, or beliefs of Greenbook.

Comments

More from Kevin Gray

Responsible AI: Balancing Innovation with Ethics

The Prompt

Responsible AI: Balancing Innovation with Ethics

Discover the world of Artificial Intelligence and unravel the confusion of basic concepts. Explore distinctions between pattern detection and generati...

Kevin Gray

Kevin Gray

President at Cannon Gray

The Impact and Ethics of Artificial Intelligence

Research Technology (ResTech)

The Impact and Ethics of Artificial Intelligence

Marketing scientist Kevin Gray asks Dr. Anna Farzindar of the University of Southern California about the impact and ethics of Artificial Intelligence...

Kevin Gray

Kevin Gray

President at Cannon Gray

Where is Marketing Data Science Headed?

Where is Marketing Data Science Headed?

Some marketing data science directly competes with traditional marketing research areas and many marketing researchers may wonder what the future hold...

Kevin Gray

Kevin Gray

President at Cannon Gray

Data And Oil

Research Technology (ResTech)

Data And Oil

The importance of pursuing the discovery and understanding of data.

Kevin Gray

Kevin Gray

President at Cannon Gray

ARTICLES

Moving Away from a Narcissistic Market Research Model

Research Methodologies

Moving Away from a Narcissistic Market Research Model

Why are we still measuring brand loyalty? It isn’t something that naturally comes up with consumers, who rarely think about brand first, if at all. Ma...

Devora Rogers

Devora Rogers

Chief Strategy Officer at Alter Agents

The Stepping Stones of Innovation: Navigating Failure and Empathy with Carol Fitzgerald
Natalie Pusch

Natalie Pusch

Senior Content Producer at Greenbook

Sign Up for
Updates

Get what matters, straight to your inbox.
Curated by top Insight Market experts.

67k+ subscribers

Weekly Newsletter

Greenbook Podcast

Webinars

Event Updates

I agree to receive emails with insights-related content from Greenbook. I understand that I can manage my email preferences or unsubscribe at any time and that Greenbook protects my privacy under the General Data Protection Regulation.*