Today we have a guest editorial as Steve is on vacation.

I love Big Data twaddle. ‘Big data will become a key basis of competition, underpinning new waves of productivity growth, innovation, and consumer surplus’, ‘Big Data helps drive efficiency, quality, and personalized products and services, producing higher levels of customer satisfaction and experience.’,  ’ In an era of dramatic technology changes, the rise of big data analytics has certainly been one of the most influential’,  ‘Big Data: A Revolution That Will Transform How We Live, Work and Think’. Yes, it’s horsefeathers, and I could go on quoting this stuff for ages, giggling between breaths.

I love this caffeine-talk, but it is wrong to dismiss it entirely as the babble of over-excited marketeers. It is a great change to be able read so much enthusiasm within the industry. They are excited about  ‘predictive’ technologies that enable us to find significant trends from the data we already have using techniques  that are by no means new, and which have little to do with the products now heavily marketed as ‘Big Data’.

I reckon that we should capitalise on the buzz about big data, by steering it gently but firmly to reality.  For years, we in the database industry have struggled to explain to businesses the great value of multivariate statistical techniques, regression and predictive modelling on data. Suddenly, the industry understands the value of this sort of  magic  but attributes it to the wrong wizard. We have the tools already. We don’t actually need huge datastores  to keep all the data once we’ve done the preliminary aggregations. The most spectacular successes of statistical modelling have used relatively small data sets, but the right data, and the tools to provide these ‘insights’ have been around for years and are freely available.  R and SQL Server fit together remarkably easily, and I imagine that there are only two things that have restrained us in the past. Firstly has been a previous lack of management enthusiasm for it and secondly  has been our lack of training in statistical regression analysis and multivariate analysis. It is up to those of us who do business Intelligence to wise up to what is really fuelling this wave of enthusiasm and exploit it by showing the industry that  it can be delivered without having to open the big wallet.