Earlier this week I read an article in the McKinsey Quarterly from Brad Brown, Michael Chui, and James Manyika entitled Are you ready for the era of ‘big data’?. It talked about “Radical customization, constant experimentation, and novel business models will be new hallmarks of competition as companies capture and analyze huge volumes of data” and got me thinking about the ways in which the emerging bigdata platforms are changing, in both subtle and disruptive ways, how we look to understand our business.
The particular term that resonated with me was constant experimentation as this perfectly encapsulates my experience. No longer having to wait hours or days to process millions of rows of data means that I can now test out a plethora of hyphotheses and just throw away the ones that don’t work out. It’s just so liberating!
Social analytics is one of those areas which is both blessed and cursed with bucket-loads of data. While the volume and variety gives you the most amazingly rich set of data, that same volume can be a total pain in the head. A couple of years ago I did my first such experiment where I used Hadoop with IBM BigSheets and IBM Content Analytics to analyze Twitter. The question I wanted to ask in this first experiment was “What do people want?”. The project was a lot less painful than I had anticipated and by the end I was completely hooked. For more info, check out BigData Analytics Made Easy.
Then earlier this year I decided to step it up a notch with a new question, “Can we identify what people are interested in , and then use that as a lens into Social Media?”. Now as you can imagine, this is where the experimentation was really critical as this question is a complex aggregation of many smaller questions that all needed to be answered. By using IBM BigSheets — which presents some amazingly simple spreadsheet paradigms for manipulating bigdata — I was able to rapidly iterate through a vast amount of data incorporating a variety of algorithms through the BigSheets macros. At one stage I was slicing and dicing a spreadsheet with more than 200 million cells. It was just mind boggling, but great fun (for total lunatics). For more info, check out Integrating Social Media into Business.
What this project did teach me though was the importance of experimentation, and for me that is the most exciting and liberating aspect of the new bigdata tools and platforms which we now have at our fingertips.
Marie's Ramblings & Ruminations 
Leave a Reply