Great title for a post, isn’t it? Seems foolish, lazy, or worse to plan for failure, right? Yet, in the real world, stuff fails all the time. Sometimes we could have done more, sometimes we can’t afford to do more, sometimes it’s just the way things are. As DBA’s we’re usually thinking about backups and server failures, maybe our share of the disaster recovery plan, and of course we get the thrill of fixing mistakes (usually others, but sometimes our own!). We tend to do pretty well on the mechanical part, sometimes not so good on the communication side. Sharing why and how long and can we or should we take steps to prevent a problem are all important, and worth building a distinct process around them.

I was reminded of this today as I went to get iced tea at Panera. The Dr. Pepper button on the soda dispenser had a hand written note that it was ‘out, sorry!’. I don’t drink Dr. Pepper, and even if I did, a long explanation of why probably wouldn’t cheer me up, and unless going to be fixed in the next two minutes, when probably wouldn’t interest me either. Fun to imagine behind the scenes; did the manager/stock person forget to re-order, did the soda guy call in sick on his route, will they start keeping ‘extra’ on hand just in case? Or is just ‘we ran out’ and it will be here later today, is that good enough?

When I saw that hand written note it was good for a smile, because I took the picture below quite a while back at Chipotle for a blog post to be written some day. They planned for failure, handled it, and threw in some self-deprecation too.

Messaging matters. Think about what you’re showing your customers when a failure happens. Imagine the tables being turned, would you think “excuses”, “spin”, “denial”? Messaging about failures requires letting go of pride and acceptance of risk and consequences. Strangely, the more you deal with it calmly and directly, the less severe the consequences.