The promise of Big Data; Fact or Fiction?

Last week I read a couple of good articles both bemoaning the hype that is bigdata. The first was from Matt Fates of Ascent Venture Partners with his article Big Data: Love the Mission, Hate the Moniker and the second from Tim Leberecht of NBB with his Why Big Data Will Never Beat Human Intuition. The reason the combination of these two articles stuck in my head is that they both articulately highlight, from opposite directions, what I believe to be a serious perception issue for analytics; one that is the result of the incessant use of the increasingly meaningless bigdata label being thrown around as though it was the panacea for all business problems.

And therein lies the problem… The growing popularity of the term “bigdata” has been a mixed blessing for us in the analytics space. On the one hand it has put analytics top of mind and on everyone’s agenda, however on the other hand it has made the consumers of analytics extremely suspicious of the promise (or over-promise). Tim comprehensively describes some of his concerns with big data and while I don’t agree with all of them, at least in their entirety, I do appreciate his frustration with all the hype.

While I’m a physicist by education, I’ve always had a strong affinity for the humanities and have worked closely with humanists in recent years sitting on advisory boards for the Digital Repository of Ireland at the Royal Irish Academy and the Digital Arts & Humanities PhD Program at Trinity College Dublin. The reason I mention this is to make it clear that I totally relate to Tim’s perspective “How much (or little) space do we leave for creativity and human expression if we equate better living with better algorithms?”.

However we don’t want to throw out the baby with the bath water. We need to be careful that we don’t let skeptism blind us to the business value of analytics, particularly the new types of analytics targeted at the “systems of engagement” which are actually trying to bring the humanity back into the analytics process, making them people-centric. A little bit of skeptism is a great thing in that it forces the business to truly understand the models being applied, and the analytics providers to create analytics that is understandable for the business. And if your analytics vendor can’t do that for you, then kick them to the curb :) We (analytics providers) need to make our analytics more transparent and need to be building analytics that has the flexibility to integrate human intuition and perspective; both in terms of how the analytics models are built (maths only gets you so far) and how its consumed by the business.

A few years ago I was doing an analytics project where I had to build a model that would help the business predict the outcome (and cost) of insurance claims by integrating all the case histories. Due to the really short time allocated for building the model, and the limited amount of annotated training data, I decided to “cheat”. I’d interview the most successful claims handlers in the business and use their intuition to bootstrap my analytics model. It was amazing the type of insight they had in their heads; in some cases it might be a particular property (like the injury type), the order in which certain things happened (property damage getting claimed before the personal injury), or the location (some counties are more conservative than others). This interview process was not only extremely productive, but also forced me to take a development approach that prioritized the integration of human feedback.

Analytics, if done right, is supportive of human intuition and its most definitely not an either/or situation.

Some related posts:

4 Comments to “The promise of Big Data; Fact or Fiction?”

  1. I actually love the moniker, but I think there is a danger in falling back on the man versus machine arguments that the Matt and Tim positions characterise to an extent. It seems to me that this thing called Big Data is not just simply a larger version of small data and/or conventional data analytics, nor does it represent an assault upon intuition. Rather, what is happening is a fusing of the two – one the one hand you could say that intuition is becoming quantified, or you could say that analysis is being socialised. In terms of the impact on society, this could be great, or it could be terrible.

    The problem is that I don’t see enough analysis of this new thing that itself is sufficiently new, rather than a re-heat of old arguments and ideas. I recently told a gathering of communications folk at the EU Council that the algorithm is the most powerful tool for social control invented since the sword. I also told them that no-one has any real handle on how to control the usage of this tool. Current EU thinking about anonymity is both ridiculous and futile – again an attempt to pull old ideas into a new space.

    Coming back to the moniker – I don’t think we have any real conception of the nature of its bigness, we tend to focus on the data bit (which is why I like the moniker and also its alliterative similarity to Big Brother, which cannot be a co-incidence). It means that the name of the game is adding more hay to the haystacks, not looking for needles within them. No data is inconsequential (can be considered as hay in the traditional sense).

    Anyway – have just published a book which deals with some of these issues to an extent. It also references the discussion we had last year at Marketing Week Live about actionability of data and social permission (chapter 6).


    • Thanks for the comment Richard :-) I particularly like & agree with your point about “big data is not simply a larger version of small data” but rather that its a completely new beast. When I think of BigData I think less of the “big” and more of the “new” types of analysis which have the ability (because of the “big”) to fuse the personal opinion with the data-driven one.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: