Last week I read a couple of good articles both bemoaning the hype that is bigdata. The first was from Matt Fates of Ascent Venture Partners with his article Big Data: Love the Mission, Hate the Moniker and the second from Tim Leberecht of NBB with his Why Big Data Will Never Beat Human Intuition. The reason the combination of these two articles stuck in my head is that they both articulately highlight, from opposite directions, what I believe to be a serious perception issue for analytics; one that is the result of the incessant use of the increasingly meaningless bigdata label being thrown around as though it was the panacea for all business problems.
And therein lies the problem… The growing popularity of the term “bigdata” has been a mixed blessing for us in the analytics space. On the one hand it has put analytics top of mind and on everyone’s agenda, however on the other hand it has made the consumers of analytics extremely suspicious of the promise (or over-promise). Tim comprehensively describes some of his concerns with big data and while I don’t agree with all of them, at least in their entirety, I do appreciate his frustration with all the hype.
While I’m a physicist by education, I’ve always had a strong affinity for the humanities and have worked closely with humanists in recent years sitting on advisory boards for the Digital Repository of Ireland at the Royal Irish Academy and the Digital Arts & Humanities PhD Program at Trinity College Dublin. The reason I mention this is to make it clear that I totally relate to Tim’s perspective “How much (or little) space do we leave for creativity and human expression if we equate better living with better algorithms?”.
However we don’t want to throw out the baby with the bath water. We need to be careful that we don’t let skeptism blind us to the business value of analytics, particularly the new types of analytics targeted at the “systems of engagement” which are actually trying to bring the humanity back into the analytics process, making them people-centric. A little bit of skeptism is a great thing in that it forces the business to truly understand the models being applied, and the analytics providers to create analytics that is understandable for the business. And if your analytics vendor can’t do that for you, then kick them to the curb :) We (analytics providers) need to make our analytics more transparent and need to be building analytics that has the flexibility to integrate human intuition and perspective; both in terms of how the analytics models are built (maths only gets you so far) and how its consumed by the business.
A few years ago I was doing an analytics project where I had to build a model that would help the business predict the outcome (and cost) of insurance claims by integrating all the case histories. Due to the really short time allocated for building the model, and the limited amount of annotated training data, I decided to “cheat”. I’d interview the most successful claims handlers in the business and use their intuition to bootstrap my analytics model. It was amazing the type of insight they had in their heads; in some cases it might be a particular property (like the injury type), the order in which certain things happened (property damage getting claimed before the personal injury), or the location (some counties are more conservative than others). This interview process was not only extremely productive, but also forced me to take a development approach that prioritized the integration of human feedback.
Analytics, if done right, is supportive of human intuition and its most definitely not an either/or situation.
Some related posts:
- Data Scientist; Too narrow a definition?
- Software Industry badly needs an injection of Diversity!
- The Future of Business Analytics; from Transactional to Interactional
Marie's Ramblings & Ruminations 