One of the finest songs of the sixties had the following lines …

"Sitting one day by myself,

And I'm thinking, "What could be wrong?"

When this funny little Hedgehog comes running up to me,

And it starts up to sing me this song.

Oh, you know all the words, and you sung all the notes,

But you never quite learned the song", she sang.

"I can tell by the sadness in your eyes,

That you never quite learned the song".

(Mike Heron: The Hedgehog's Song)

I once spent a fascinating couple of years auditing and inspecting the databases of Her Majesty's Government. I often hummed the Hedgehog Song to myself as I stared at the database scripts and listened to DBAs in charge of the databases that affected the lives of millions of people.

I've seen many databases; a few great ones, some good'uns and many gor'blimey ones. After a while, it was easy to detect the difference. Mike Heron's sad song about love applies just as appropriately to database design. There is a world of difference in knowing the words of relational theory, and being able to hum it, but the practice of doing it properly is another stage.

The well-constructed relational databases just seemed to work. They didn't require endless tinkering and maintenance. Tables were well normalised, narrow, appropriate, and ingenious. Constraints and keys abounded. The data model fitted the business like a glove. The SQL was simple because it just needed to declare what result was required, no hints, no fancy stuff, and only rarely extra indexes. The effort was focused on maintaining an immaculate, normalized representation of the organization's data. On the rare occasion that a query was slow, the dev teams assumed that the fault lay in the data model, and they'd fix it.

There was one such database I knew that was responsible for the clearing of cheques for a major retail bank. It processed millions of business transactions a day. The team, a handful of people that created and maintained it, were nearing retirement so two major IT projects, involving hundreds of bright young developers, aimed to replace the system. Both failed. The original team became used to being called away from tending their roses to maintain the system that kept the organization functioning.

As always, it isn't so much the algorithms but the data structures that determine how effective a relational database will be. If your database consists of a few poorly designed "Godzilla" tables, it is much harder to re-engineer them without the lights flickering, raised voices, and the sounds of approaching sirens. How can one justify iteratively developing a database model when it is easier to keep things running by hints, heuristics, and hardware? My experience tells me that disguising the problem in a relational database always ends badly, because, by doing so, it escalates complexity, and that is a scary trend for any IT system. Instead, narrow, well-designed relational tables will support nimble, iterative development, but it requires knowing how to sing the song.