I posted a note on Twitter/X with this quote: “The content updates had not previously been treated as code because they were strictly configuration information.” This is from testimony given by Crowdstrike to a US Congressional committee in trying to explain how they grounded much of the airline industry a few months ago. That was a mess of a situation, and apparently, the vendor didn't think their configuration was part of their code.

That's an amazing viewpoint to me. The fact that any developer or manager thinks that their configuration data isn't a part of their code is worth testing. Yet, I see this attitude all the time, where developers, QA, managers, and more think that the code is the only thing that changes or doesn't change, ignoring the fact that there are configuration items that affect the code and need to be managed appropriately. Certainly, if the config data were in enums rather than in a file or database they'd feel differently.

I think part of the reason that people try to ignore config data is that it is hard to manage. Often config data might change between dev, test, and prod. Dealing with that, and testing appropriately is hard. I haven't ever seen a good solution for getting data into an environment the first time. That's the hard part. Once the data is there, you can use it as a token where it is needed, and hopefully, the value has already been tested. At the very least, you can test how that data affects that environment.

I am glad to see Crowdstrike publicly recognize that they need to dogfood not only their code changes but also their config changes. However, for a company that hasn't shown a rigorous engineering approach, I suspect they'll test very simple and basic config changes and not necessarily do a good job of carefully testing a variety of potential problem vectors. That takes work, and excluding config data from testing is a sign (to me) of a technology group trying to avoid doing too much work. It's likely more management and leadership than technology workers, but the entire organization is showing signs of shortcutting good engineering.

My view is that developers should be free to experiment and try lots of things, and have a lot of freedom on how they build software. I think the same thing for infrastructure people as well. However, as we start to move our changes towards production, everything should be in code, version-controlled, and promoted through PRs. In other words, get everything stored as code and nothing gets changed outside of development. It only gets approved to move forward or rejected, after it's well tested.

That's a tough process to implement, and one many companies don't spend the time doing, but for those that do, they end up deploying many  fewer bugs.