February 16, 2014 at 9:38 am
I was watching a video a few days ago where I think it was Scary DBA remarking off-hand how some managers don't really understand the differences between programmers and DBA/DBD.
My paraphrasing of the argument is that with programming you can be reasonably confident that whatever was on the developer's machine will end up in the build which ends up on the user's computer. But with databases you can neither just "replace one section of code with another" because there is data in the equation, and also that DBAs can jump into production databases to make changes that immediately disappear from sight.
But I wanted to hear more about that. Here are some of my ruminations:
- This is all assuming your database is in source control at all. I feel like even this isn't enough; unless you can validate your source control 100% represents your database (i.e. schema and procedures) then it's almost as bad as not having it at all.
- You might have just one build of software. But many "pseudo-builds" of customer databases with slight differences. Because unless you've been very diligent in your procedures most builds are applied as changes to existing structures, and all kinds of things can happen for drift to occur over time. (e.g. the above hacking, "urgent" requests, failed builds, missed builds, builds not in source control).
- A programmer creates the data once and often with little concern for its quality or longevity. But the DBA/DBD is the caretaker for it every day after. Backing it up, ensuring it continues to meet validations, performance tuning it, and eventually archiving it (or deleting it entirely).
Sometimes I do feel the pressure of management viewing DBA/DBDs as they do programmers; that is they are there to add value to the business through a bottom line - something that can be charged to customers. And so I often have to stand my ground (with all the stress etc that is involved in that).
It is great that companies want to increase revenue, and they have staff to do those things. But if nobody is looking after backups, improving data quality by finding and fixing problems, and generally improving processes to remove technical debt... then it's not like I can just switch off my, "I care about this" feeling and go write a revenue generating report.
But I think it comes down to understanding the differences between DBA/DBD and software development and being able to explain it in a way management can understand, but without waiting for disaster to strike first. So, can anyone add some experience to the discussion?
February 16, 2014 at 6:12 pm
Cody K (2/16/2014)
My paraphrasing of the argument is that with programming you can be reasonably confident that whatever was on the developer's machine will end up in the build which ends up on the user's computer. But with databases you can neither just "replace one section of code with another" because there is data in the equation, and also that DBAs can jump into production databases to make changes that immediately disappear from sight.
It doesn't work that way in my shop and shouldn't work that way in any shop. Just because a DBA is directly related to the "all powerful OZ", doesn't mean they have the right to change any code without it going through testing and redeployment.
In our shop, all T-SQL goes through a DBA peer review. I use these peer reviews as time for 1-on-1 mentoring which has really improved the quality of the original work and has greatly reduced rework and performance problems especially as a database scales up. The code then goes through QA and any changes to the code must be re-reviewed by the DBA (me, if you haven't guess that already) no matter how trivial the change may be.
Once code has successfully QA'd, even I can't change it without it going through a peer review with the Developer and retest by QA. Once it's in production, NO ONE can change it without it starting the process all over again.
While that may seem slow and arduous to many, rework is 8 times more expensive and I don't need to tell you how costly a "mistake in production" can be. Because of the peer reviews, we've cut QA and rework time down substantially. The little bit of extra work up front has saved us shedloads of time lost on even trivial problems.
And, yes. If you've properly written your DB code, you can, in fact, just replace one section of code with another and Bob's your Uncle. Good database code is not abstraction resistant.
For shops that have made the mistake of allowing Developers and DBDs to promote code directly to production themselves, thank goodness a DBA can intervene. I do blame some of that mistake on the DBAs themselves because they've not made the need for Peer Reviews and QA understood beyond any reasonable doubt.
--Jeff Moden
Change is inevitable... Change for the better is not.
February 16, 2014 at 7:28 pm
It doesn't work that way in my shop and shouldn't work that way in any shop.
This is really what my post is about. Changing a shop from the wild west to a proper methodology.
While that may seem slow and arduous to many
To many, but not to me.
And, yes. If you've properly written your DB code, you can, in fact, just replace one section of code with another and Bob's your Uncle. Good database code is not abstraction resistant.
I agree up to a point, but absolutely not in a shop that isn't using source control properly.
I do blame some of that mistake on the DBAs themselves because they've not made the need for Peer Reviews and QA understood beyond any reasonable doubt.
I think the managers understand the importance of it because they ask for it. But then once you start working on implementing that process, and you start getting asked why you're working on it (because it was requested?!) and the talking begins about how disruptive and unnecessary the changes are... I have a problem. Management backs down because they get told "manual is good enough".
February 16, 2014 at 8:14 pm
Cody K (2/16/2014)
I think the managers understand the importance of it because they ask for it. But then once you start working on implementing that process, and you start getting asked why you're working on it (because it was requested?!) and the talking begins about how disruptive and unnecessary the changes are... I have a problem. Management backs down because they get told "manual is good enough".
I believe you have just defined the terms "Wild West Programming/Managment" and "Perpetual Repairs". 😀
--Jeff Moden
Change is inevitable... Change for the better is not.
February 16, 2014 at 8:33 pm
Thus the post. I imagine because the managers understand things from a software perspective; source -> build -> application, and need some cajoling to understand database -> manual process that works most of the time with lots of upkeep -> output.
I'm sure others have been in this situation and I want to know how they put these things into terms "software managers" can understand. And part of that must be picking out how databases are different to software.
I'm sorry this is a very roundabout way of asking a general question; the exact kind that I hate, but of course I'm trying to keep details minimal too.
February 17, 2014 at 7:41 am
I'm going to take a guess and assume that you're not talking about my session called "Best Practices for Database Deployment" because I get into a lot of what you're talking about there. I make a huge point of saying that DBAs are not the special snowflakes that they think they are and that process, whatever it might be, absolutely has to apply to them equally. And yeah, I do believe that it all comes back to source control because I think you have two known states for your systems, what's in production and what's in source control. The more those are synchronous, the more you can control your deployment processes and automate them to make them faster and more reliable.
In short, I don't think DBAs are good at the development/deployment aspects of our job. We're great at the "protect production" part, but let that override the other. We need to do both.
"The credit belongs to the man who is actually in the arena, whose face is marred by dust and sweat and blood"
- Theodore Roosevelt
Author of:
SQL Server Execution Plans
SQL Server Query Performance Tuning
February 17, 2014 at 9:03 am
Grant Fritchey (2/17/2014)
I'm going to take a guess and assume that you're not talking about my session called "Best Practices for Database Deployment" because I get into a lot of what you're talking about there.
Hi Grant! That's exactly the one I'm talking about. I kind of wish it had diverted into a whole talk about management instead.
I make a huge point of saying that DBAs are not the special snowflakes that they think they are and that process, whatever it might be, absolutely has to apply to them equally.
Well you're preaching to the choir and it's all common sense to me. But I think you've misunderstood the direction I'm coming from. Argh, so frustrating for us both 🙂
From what you've said I almost feel like you are implying that process is handed down on tablets from management, where IME (probably much smaller shops than yours) it's more likely to be something that bubbles up from employees and only after a bloody turf war against the old guard of manual processes.
Management gets stuck in the middle. The elders are saying everything is fine when it clearly isn't. The relative newbies are obviously just trying to play with new *cough 14 year old cough* technology.
And I think it comes down to management not understanding that it's quite difficult to build a modern application without source control because every build is a complete state. It's very easy to run a database development and deployment process manually because all those databases are maintaining their own state and you're only sending out many hand written error-prone scripts.
And so I wanted some more ideas on explaining things in that way to management, to garner support, against those who think manual processes are good enough.
I sure hope I haven't dug a hole now.
In short, I don't think DBAs are good at the development/deployment aspects of our job. We're great at the "protect production" part, but let that override the other. We need to do both.
I care about both so we're on the same page there. I swear I'm not an ingrate 😛
February 17, 2014 at 9:25 am
Ingrate!
Kidding.
Interesting take. I sure don't think that management hands down tablets for us. In fact, the opposite. We should be telling management what we're doing and how we're doing it. It's more that I see developers as having a good/great/decent handle on deployment and deployment processes and we, as DBA/DBDs need to figure out how best to make our processes, which are very different from what the appdev people can do, incorporate into their processes.
Explaining that to people... believe me, I'm trying. I still haven't found what I think is the magic words to really get people to buy into the concepts. I've been the most successful with the stuff you saw (and that's the "polished" version <shudder>).
I think, to a very large degree, you have your finger on the issue. "We've always done it this way" is extremely difficult to overcome, especially when people don't recognize the pain they're experiencing.
"The credit belongs to the man who is actually in the arena, whose face is marred by dust and sweat and blood"
- Theodore Roosevelt
Author of:
SQL Server Execution Plans
SQL Server Query Performance Tuning
February 17, 2014 at 9:31 am
Phew!
You know whenever I talk with you famous people I get nervous and feel I might have accidentally destroyed my career for thinking out loud 😀
Viewing 9 posts - 1 through 8 (of 8 total)
You must be logged in to reply to this topic. Login to reply