This month’s TSQL Tuesday is organized by Kennie T Pontoppidan(t) – the topic is ‘Daily Database WTF‘ – or a horror story from the database world. As someone who has worked databases for nearly two decades, there are several of these – I picked one of my favorites. During my early years as a DBA, I was doing some contracting work for a government agency. At one of these agencies I encountered a rather strange problem. Their nightly backup job was failing on a daily basis. The lady DBA in charge was very stressed trying to solve it. It was a TSQL script written by someone who was no longer with them. According to their process, every failure had to be extensively documented on paper and filed for supervisors to review. So she had a file on this particular job failure that was bigger than the yellow pages, on her desk .She tasked me with finding out the reason for this job failure, as my first assignment.
I dug into the script – it was simple – it pulled an alphabetical list of databases from system metadata and proceeded to back them up. It didn’t do this one simple thing – leave TEMPDB off the list. So when the backups got down to TEMPDB, they promptly failed. Now as a smart person – I should have just communicated this to her and had it fixed quietly. But, I was young and rather hot headed at that time. It amazed me that a DBA with several years of experience did not know that TEMPDB cannot be backed up. So, I waited until the team meeting the next day. And when the said job failure came up again – I said that I knew the reason and stated why. I also added that this was a ‘very basic thing’ that junior DBAs are supposed to know. I was stopped right there. It did not go down well. Her face was flaming red because a consultant showed her up in front of her boss in a bad light. She said she would talk to her boss and collegues the next day (several of whom were nodding their heads disapprovingly at me) and meeting was abruptly adjourned.
On my way back, I heard several people whisper that she does not like or do dissent, and I did a very unwise thing by pointing out something against her in public. I told myself that working with such a person would not be the right thing for me. When I came in to the office the next day, the HR girl was waiting at my desk. I was escorted to her office and told that my behavior was not acceptable coming from a consultant, and that my contract was terminated. I still recall being walked to the door with my jaw half open. Before she shut the door on me I said ‘there are still issues with that script, if you want to know’. Well sure enough they didn’t, and I went home. I really, honestly felt amused and free, despite the fact that I had no job to pay my bills.
In another week’s time, I had found another gig and moved on. I heard later about the issue I had not talked about. The databases that lined up after TEMPDB alphabetically were never backed up. In months. They had had a massive system failure and found out that they had no backups to recover from it. I don’t know if the lady in charge or her friends suffered any consequences, it was a government job after all and they could not be easily fired. But I do know that they found out the hard way and somebody probably paid for it.
I always narrate this incredible story to people who get fired for things that they did right, and also to watch your back in politically sensitive situations.
