Software Updates, Outages, Processes, and Protocols

This past Thursday, February 22, AT&T had a major outage on their U.S. network. For upwards of 10 hours, hundreds of thousands of customers could not make phone calls, send or receive texts, or use mobile data for apps or browsing websites. Aside from not being able to communicate as normal, it also appeared to knock the ability of users to access emergency services like 911, raising the level of alarm. It was such a major communications event that the FCC and the US Cybersecurity and Infrastructure Security Agency are now involved in the investigation.

Thankfully the initial investigation does not appear to show any cyber-attack as the cause. Instead, the response from AT&T on Friday was that the incident was caused “by the application and execution of an incorrect process used as we were expanding our network…” Inside sources have told some news outlets that it was specifically a software update that didn’t complete correctly and took essential equipment offline. In other words, somebody (or team) didn’t follow the playbook. All I could think about on Thursday in relation to this story was #HugOps. J

Here's the thing. In more than 20 years of IT operations, software and database development, and building multiple SaaS applications, I have personally never had to worry about a mistake, even a failed upgrade, putting people’s lives in danger. Although I, and my teams, have always striven to provide an excellent experience for our customers at every level, nobody lost their job if utility bill data couldn’t be entered into the application for a few hours. When we did fall short of our goal and our customer SLA was breeched, it was always a learning opportunity to prepare better for next time.

The longer you work in this field, the more you recognize that preparation is key. Checklists, playbooks, and planning for how to react when the unexpected happens are essential skills to learn and build within a team. I’m positive that the folks at AT&T had all of that, and still a major outage occurred. Thankfully it appears to have been remedied in less than 10 hours, which honestly feels like a speedy recovery considering the size and scope of the network they deal with. Jokes about DNS, BGP, or an expired certificate aside, it was a good reminder for us all that thinking about, and planning for, the unexpected is always a part of our jobs. I mean, none of us want to end up in the headlines when data loss occurs, accidentally or not, right?

If you were affected by, or aware of, the outage this past week, what did it make you think about in relation to your job? Do you see opportunities to prepare in a new way the next time you release and update or modify the database schema? Give it some thought if you haven’t. And remember, in times like Thursday, it never hurts to send out some #HugOps.

Ryan Booz

Join the debate, and respond to the editorial on the forums

Flyway can scale easily to enterprise-scale database systems, even if they involve a mix of relational database systems, are cloud-based, containerized or involve complex authentication. This article demonstrates how we can use Flyway Teams to do a single-batch, multi-database migration, comprising SQL Server, Oracle Cloud, PostgreSQL, MySQL and SQLite databases.

Recently I have been working with some queries in ...

It’s been great to see the interest and particip...

For this month’s TSQLTuesday, I asked y’all to...

The use of multiple database platforms is on the rise, however, managing them can be challenging, with issues such as data integration, monitoring and a requirement for greater skill diversity. So how can you thrive in this environment? Join industry experts: Joshua Higginbotham, Senior Architect at Centric Consulting; Mri Pandit, Senior Manager at Navy Federal Credit Union; and Redgate Advocate Steve Jones on March 19th, as they share their insights on managing the multi-database platform landscape.

Jess Pomfret makes an announcement: Replication, everyone loves to hate it, but it’s been our most requested feature as far as adding commands to dbatools.…

This article covers how to use Azure Data Factory ...

Looking at my calendar this morning, I realized it was time for the Simple-Talk editorial to go in the newsletter. As I searched my lists of possible topics I had accumulated, none suited how I felt. Then it hit me, “duty”. This editorial needs to be written, and it is my duty to make that happen.

Reading Time: 7 minutes In this post I want to co...

Muhammad Ali shows off another way to back up and restore data in Postgres databases: Barman is a production grade tool for managing the backup…

When it comes to the development and operations (DevOps), one thing stands out as a critical aspect and that is troubleshooting. The primary goal of a DevOps team is to ensure that the product experiences zero to no downtime because every moment is crucial. Therefore, smooth delivery and uninterrupted uptime are paramount. To achieve this, DevOps teams monitor the product’s performance using logs, and databases, like MySQL, are no exception to this practice.

What SQL Server’s Query Optimizer Doesn’t Know...

I’m The Face A lot of the time, the answer to pe...

A Trivial plan is created when SQL Server really d...

Ever wanted to connect to a data s...

I have published videos and articles before about Lakehouse maintenance. In this article I want to address a missing point for a lot of Fabric administrators: How to do maintenance on multiple lakehouses that are located in different workspaces.

Presently sponsored by: Report URI: Guarding you f...

In software development the concept of feature tog...

Rich Benner builds a list: One potential issue here is the lack of time to delve really deeply into each programming language at one’s disposal.…

Welcome to the first of three blog posts about the...

Flyway can scale easily to enterprise-scale databa...