Q&A: Dealing with Thousands of Databases (Part 3)

I’m going to punt to my “A Day in the Life” series here.

I’ve only been at my current job for about 2 1/2 years, but I can speak to the shorter intervals. I’m going to be intentionally vague in spots here as I don’t want to disclose too much.

One Month: I didn’t expect to be terribly productive for my first 2-3 months as I knew there would be a difficult learning curve as I got up to speed with the environment, the business, and the people. Reality? I had the day-to-day stuff picked up within the first month and added on from there. One of my goals was to quickly become proficient enough to take my share of on-call and the software release process (which we were doing weekly at the time) to take some of the burden off my colleague’s shoulders. I was the sole on-call resource for two straight months with no relief at a previous job. As a result, I knew she had been feeling a lot of pressure as the lone DBA for several months before my arrival. As her teammate, part of my job was to share that burden.

6-12 Months: Expectation: Upgrade to SQL Server 2016, start investigating a move to the cloud, find all kinds of things to tweak/tune/optimize, help developers make changes to how they access the database to make things work better.

Reality: Within a week or two of starting, the technology roadmap took a sharp turn, and further discussions in that vein had me seriously wondering if I’d even make it through my first year (to the point of having conversations like “what did I get myself into, and how long should I try to ride this out?” with other community members). The upgrade got pushed back. Eight months in, the company was acquired. By the end of my first year a number of things changed, mostly back in the direction that I thought they would be going when I first started. I also started learning how much database-related technical debt there was and saying “hey, can you just do X?” isn’t the way to affect change. I did mange to optimize some things, but not as much as I thought (what I was able to do has ended up paying off handsomely).

30ish Months: I’m now on the track that I thought I’d be on when I started. SQL Server got upgraded. I’ve been able to work effectively with developers to clean up troublesome queries. I’m integrating my database changes into the work items for the sprints so that the QA team sees them and can properly include them in their test plans. We’ve taken advantage of new features in SQL Server to make significant improvements in queries, and watched developers do the same to make their code more efficient too. The cloud is coming. I’ve moved my mindset from “there’s no way we could do that in the cloud” to “hey, this might actually be possible, and it’ll be so cool if I can be here to make it happen!”

I think the most helpful thing that’s helped me is talking to other people in the community and using them as a sounding board. Chris Sommer (blog | twitter) started a job across the parking lot from my office about a month after I started, and we had quite a few conversations during lunchtime walks that helped me work through ideas and difficulties. Similarly, talking to folks at SQL Saturdays, local user group meetings, PASS Summit, and the SQL Community Slack channel.

Hosting this many databases on a single instance is so rare (or at least rarely talked about) that it’s hard to find websites and blogs that discuss it. There’s been lots of trial and error, lots of reliance upon tools that my predecessors have built or set up.

In day-to-day operation, it’s not terribly different from managing a smaller instance, you just work slower, things take a bit longer and when you discover that you’ve pushed something past its breaking point, you take a step back and look for a new direction to take.