One of the worst crimes in IT is to generalize from particular experience. The more experience we get, the more that black and white dogma about 'best practices' or 'patterns' fades into shades of grey. So many times, the teams who create new ways of handling, or processing data, look up from their work, and declare that it is good and suited to all conceivable purposes. Sure, it is fine for the type of data you can imagine, but what about all the other types?

In a former role, evaluating database technologies for a large multinational corporate, I was forced to check the products of an unceasing procession of optimistic entrepreneurs presenting new ways of handling data. Over twenty years ago, there was a wave of NoSQL hysteria, and a succession of suited presenters wobbled their jowls in sincerity at me and solemnly declared relational systems to be dead in the water, soon to be replaced by object databases. It did not happen, of course, because the people who developed the new generation of databases had no idea of the range of requirements of scale, resilience, security, data-flow, and the sheer interdependence of corporate data. There is a world of difference between the way of responsibly handling test data, departmental data, private research data, financial data, personal data, manufacturing data, and sales data. Handicapped by their hazy knowledge of the realities of data processing, none of the hopeful companies who I evaluated survived even a handful of years.

Usually, I pitied, and sympathized with, the people I met who were attempting to break the supremacy of the relational model for handling corporate data. I explained the burden of support and training that the introduction of a new technology entails, the sheer scale of the data it would be required to accommodate, the almost science-fiction demands on performance. I waxed eloquent on the statutory requirements for data, the corporate rules, the conditions for putting database systems into production, the complexity of data processing. I'd dip into my fund of horror stories of the consequences of leaving even a trace of doubt in the data and its integrity. I don't think I ever even bruised their confidence in their new solution.

Yes, we always hope to see a better solution for semi-structured data or for ephemeral information. All developers bridle at the restrictions of SQL Databases, but cannot help admiring their sheer ubiquity, their ability to provide data integrity and accessibility for the whole tapestry of data categories thrown up by human endeavor.

Phil Factor