PureSpectrum - Schedule A Demo
Qualtrics: Here to Help

Big Data’s Big Hype – Why Big Insights Are So Elusive

Why have we not yet seen the Big Data revolution? Where are all the Big Insights we have been waiting for?



By Allan Fromen

Big Data is big news. Not a day goes by without some article proclaiming how Big Data will solve long standing social and economic problems, in areas of education, healthcare, public policy, workforce efficiency, supply chains, and so on.

And make no mistake – Big Data is big business. IDC predicts the Big Data and Analytics market will grow 50%, from $40 Billion in 2014 to $60 Billion by 2019. Companies clearly see the potential to extract meaningful intelligence from Big Data. With innumerable sources of data, if we can only analyze it effectively, we will usher in a new dawn of actionable insights that will drive transformation, innovation, and profits.

So why have we not yet seen the Big Data revolution? Where are all the Big Insights we have been waiting for?

The problem isn’t the amount of data. In fact, the Digital Universe Study by EMC (in partnership with IDC – full disclosure) estimates that the amount of data is doubling every two years and will reach 44 trillion gigabytes by 2020. Yet this same study estimated that “less than 5% of the useful data was actually analyzed.”

So we are awash in data sets but are only utilizing a tiny fraction. Why is that?

Imagine the Marketing department wants to explore whether certain attitudinal measures drive tangible benefits to the business. Easy enough. Survey your customers and correlate the survey data with database metrics, such as number of visits, average wallet size, and so on. However, there will inevitability be numerous data issues to consider on the back end. Should we remove outliers from the survey? Does an atypical distribution of survey responses indicate an anomaly to be treated with suspicion or an important sub-segment of the market that we discovered? Our database metrics will pose an even greater challenge, as some customers will have data on certain variables and not others. Do we restrict our analyses to only customers with the full set of database metrics? If not, how do we treat missing data? What about the even greater challenge of integrating third party information?

Even in this fairly straightforward example, there are many decisions to be made. The process of cleaning and preparing the data for analysis would likely take many weeks. The challenges are exponentially greater with Big Data, as the data points are novel, numerous, diverse and – perhaps most importantly – in different formats. According to Dan Vesset, IDC’s lead for Big Data and Analytics, a data scientist spends a full 80% of their job on data preparation and cleaning. It’s no wonder Big Insights are so elusive, when so little time is spend on the actual analysis.

About a year ago, I wrote Why Big Data Will Never Replace Market Research. In that time, the enthusiasm for Big Data has certainly intensified. But what is missing from the conversation is a healthy dose of skepticism – not in Big Data’s potential, but rather in how easy it is to derive meaningful insights from multifaceted, and often unstructured data.

While the promise of Big Data is sexy and prone to attention grabbing headlines, the sober truth is that most Big Data work is boring and tedious. There is certainly no doubt that Big Data is the future. But the road to Big Data is paved with the challenging work of cleaning, preparing, and integrating systems that were designed in silos, before we understood the value of aggregating and analyzing disparate data points.  And that is why Big Insights are still so elusive.

Please share...

6 responses to “Big Data’s Big Hype – Why Big Insights Are So Elusive

  1. Allan – there are plenty of skeptics regarding Big Data; I’m one. But I think the issue of data cleaning is misleading, in the sense that once we get it cleaned up, many still don’t know what to do with it. The “let’s correlate everything and see what pops up” mentality still dominates, imbuing the data with mystical properties remaining to be discovered. I think that’s why we end up being disappointed in Big Data outcomes. A targeted question such as your example is a much better use of the data. The fact that the analyst has to think about what they are doing, maybe thinking long and hard, is not a barrier – it’s what analysts are supposed to do.

  2. Allan and Steve,

    I share your skepticism about the headlong rush to do something with big data.

    In particular, having worked in a traditionally Big Data Industry (Healthcare), the need for analytic thinking to take place in advance of data diving is a step that seems to be missed too often. Starting with an understanding of a market’s underlying dynamics as well as the particular characteristics of the data that are available (sources, cleaning, etc.) makes the difference between “correlate everything” projects and big data analyses that end up producing market insights.

  3. One contribution of market research has been the menu driven capability of many interrogation systems that are now common place in market research. Thee have been around since the early 80’s and gave the young market researcher incredible access to data even if the scale was somewhat smaller. So in essence the insights that big data promises should be no surprise to market researchers. Or more particularly, whats so new here?

    The fact that companies never interrogated their data was because few of then knew of any software that was capable of this menu driven analysis. Why didn’t they ask market research suppliers, since the technology has been there for decades?. Of course the SAP’s, Hadoops and other systems have now made it seem like this all happened in the last 5 years and that you better get on the bandwagon for breakthrough insights. .

    In a project I was involved in for a large satellite TV provider in Malaysia we learned of their inability to track churn and subscriber data. It didn’t seem like a hard problem to us, brought up on a diet of menu driven anaytics. And it wasn’t. My colleague a market researcher with an SQL background got into the data bases and had the menu driven system up and running in a matter of a few weeks. This of course put the IT department out of sorts because they had been pushing the same corporate line that this is all super hard and can only be done by some experts with their big data software. As Steve Needel said cleaning the data is a minor problem. Who are these people spending so much of their lives on data cleaning? This can be automated with tools as simple as excel.

    In my mind any good market researcher with a solid quant grounding and some knowledge of data structures and interrogation software finds this whole Big Data thing like that line from Macbeth “a tale told by an idiot, full of sound and fury, signifying nothing”

  4. So to paraphrase Hitchhiker Guide to the universe : So the answer is big data…but what is the question?

    In my mind big data sets are more for automating algorythmic marketing.

    I also question what we mean qualitatively by “insights” in the conversation about big data’s potential. We may we use the same word for tactical interesting superficial relationships…and less strategic mindset shifting deep profound understanding about people

    The big data hype is more about sustaining investment in ITsoftware . The reality will be different and have less broad impact than we think.

Join the conversation