This post comes from a conversation I had with a developer last week as well as my experiences over the weekend. My developer friend and I were hashing over war stories. He's a senior development/application architect level and he's seen his fair share of poor production deployments, one of which was finding a totally new architecture (untested) in a about to go to production deployment because a junior developer decided to show a colleague about this new, fancy technology he just learned about. If I remember the war story right, the original code had cleared QA and was about to go to production when the junior developer worked his "magic." Good thing he was intercepted because his changes revealed several things:

• The changes didn't work.
• The changes revealed the developer didn't actually understand the new technology.
• The junior developer would put a deployment at risk to obtain personal "glory."
• The build process needed significant strengthening.

This one was quite surprising to me. Having been a developer for a number of years, I wouldn't personally roll any new code into a production deployment after the initial code passed QA testing unless we found something significantly wrong and we just couldn't wait. "Just couldn't wait" is usually defined as regulatory requirement, but it could also be a significant feature release where you're trying to beat a competitor. However, in both cases I can count on one hand the number of times I've been part of a project where this kind of change was done. It should be common sense how much risk one incurs by doing something like this. But yet, that one developer did it. And as a result, the company not only lost his hours as he re-architected part of the production deployment, they lost the hours of the senior developers that then had to go in to check the code and ensure everything was rolled back properly and the right build was put forth for the production deployment.

Now to this weekend. I wasn't directly supporting a particular performance at my church, but got pulled into it because there were issues with the group's computers and they didn't know how to solve them. As it turns out, they had bought a brand new laptop with a brand new operating system they weren't familiar with (WIndows 7) and had gone from an older version to a newer version of a key piece of software (2.5 to 3.5). That key piece of software is what drives all the sound and video files for the performance. And, of course, the people involved had never used Windows 7 nor were they up to speed with the latest version of the software. In fact, they had downloaded the upgrade the night before because they wanted to switch laptops and the old version (2.5) didn't run on Windows 7. Actually, the new version (3.5) doesn't run perfectly on Windows 7, either, but apparently they were licensed for it and not the latest version (4.0), which does run on Windows 7. It runs well enough, but there are enough glitches that I would upgrade further.

At most churches this group would have went to, they would have been in some trouble. I don't mean with the operating system. Windows 7 is not that big of a deal for anyone who is a power user or works in IT; it's just a matter of actually looking a bit more carefully at a couple of screens when setting up that secondary video output because some of the navigation has changed. But that software isn't straight forward and there are places where you have to look at settings inside that software to toggle important things like desired display settings. It doesn't just conform to whatever the monitor is. The group clearly had no idea that those settings existed (I don't believe they were in the older version) or that those settings were causing all of their media to display incorrectly. We just happen to run the software they were using at the 3.5 version. So we knew what tweaks had to be made, and we were able to get them up and running. It's a good thing we were familiar with the software because I don't believe the vendor has weekend support hours and the first performance was last night (Sunday). Had that not been the case, they would have had issues putting on the presentation. I am assuming they would have dropped back to the old laptop, but they didn't feel it was "stable" but didn't elaborate as to why. I didn't bother to ask why, as I had ministry preparation that I needed to get done which coincided with the performance. And while we're not takling about putting together a new software build, what they did was essentially the same thing... throwing something new in at production deployment (in this case, a performance).

The moral of these two stories is a simple one: don't roll a change into production at the last minute which hasn't been tested. I know there are exceptions. There are always exceptions. But that's what they are supposed to be: exceptions. The rule should be everything that goes in has gone through a testing cycle. It's just the right way to do things. And it saves hours, stress, and reputations to do it the right way.