July 4, 2016 at 3:59 am
Hello all,
I am looking for help to justify my access to Production to my CIO. Sounds strange, doesnt it?
Well, it is, and I am at wits end here and feel like I am dealing with a bunch of fools :ermm:
long story short. We have an ISP provider coming in to take over the "infrastructure part of the company". But my role is not in scope (we will see). I am the only DBA in the company, and I work on both Production and non-production Databases.
This morning i was asked to document why I would need access to Production (both servers and databases). And short of "its my job to have access to production to keep it running", I am totally dumbfounded as to why they would even contemplate removing DBA access from Production instances.
Has anyone had this situation develop in their company where they have to explain why they need access to Production? To me it is like saying to a Pilot "you can fly, but you cant have a plane", you just cannot do your job.
Very frustrating....
July 4, 2016 at 12:24 pm
Patience is a virtue here. When people ask me what I do as a DBA, I frequently tell them that it's too long to explain but they'll know for sure when I stop doing it. 😛
I'm been through this totally ridiculous and immature scenario before. For some reason, people think that 3rd party teams are much smarter than the resident DBA. I've proven that wrong many a time prior to them removing my privs to production and it's usually pretty easy. Backups done by 3rd parties are normally on a silly and not-so-healthy schedule that don't meet either RTO or RPO requirements. I can't do my normal health checks (it's amazing how bad 3rd parties actually are at that), my normal job reporting, any code performance checks, nor figure out any optimizations when it comes to what actually needs index maintenance and stats maintenance to save huge amounts of nightly processing time and log file space, and I can see such things like who's causing blocking, why the blocking is occurring, and what can be done about it. I can't research improvements to avoid deadlocks nor even check for them and I've yet to see any 3rd party even monitor for them never mind make suggestions to fix them.
Remember that the goal of the 3rd party is to make money. They're not (for example) going to suggest nor even identify that a monster audit/log table could actually be partitioned so that legacy months don't need to be reindexed and you don't actually need more disk space to rebuilt the entire clustered index. They're not actually going to identify performance challenged code that has reached a tipping point in the execution plan that now causes the query to use huge amounts of memory. Nah.... They're just going to tell you that you need to buy more disk space or buy more memory or, perhaps, buy a wad of SSDs instead of correctly identifying the problem and making a (possibly) minor tweak to a bit of code to increase performance by 60X.
Yes, 3rd parties can help but there's no one more concerned about the servers and the data than the resident DBA. If they remove that person's privs in prod, they're crippling one of the best resources they have that will measure, anticipate, and actually prevent problems from occurring rather than fixing them after they've happened. And no 3rd party will have an esoteric knowledge of the databases where data issues can be solved in a relative flash.
And, who the hell is going to promote code. The 3rd party? Some automatic system? BWAAAAA-HAAAAA-HAAAAA-HAAAAA!!!! Good freakin' luck with that! Humans make mistakes but if you really want to screw something up as quickly as possible, let a computer do it and then remember who wrote the code for THAT! HUMANS!
Ok... what I'd do is sit down and carefully write a list of all the things you do as a "Production DBA". It's going to take some time because DBAs tend to take a lot of what they do for granted or just a part of the daily nuances. I'd also include your passion and esoteric knowledge of the data, code, and schedule and then explain that the 3rd party just doesn't and won't ever have anyone with that type of knowledge because the "next body on duty" is the one that will get assigned to the job.
Also, I worked for a company that used such a 3rd party... that 3rd party didn't even have someone that knew how to get the current date and time using T-SQL. Ask what the qualifications of the 3rd party DBAs are.
I can sometimes see moving to offsite tin but I can't see relying solely on 3rd party "expertise" (which can be surprising low) that doesn't necessarily have a vested interest when it comes to the database, safety, and performance. You won't believe the excuses they'll come up with and how much they'll try to shift the blame to you guys, your code, and your database design when something goes wrong.
And ask them how many times a year they'll actually do a test restore of your databases to make damned sure that they can be restored. I restore my "money maker" every bloody night and my RPO is less than 15 minutes. I've reduced my RTOs from more than 10 hours to around 10 minutes on many of may larger databases. Can they do the same? They probably could but it will cost you... and cost you... and cost you. 😉
Now, if they insist, you have two choices... 1) become a monster Application DBA or 2) sharpen up your resume and start looking. People are dying out there trying to find good DBAs. You were looking for a job when you found this one, right?
--Jeff Moden
Change is inevitable... Change for the better is not.
July 4, 2016 at 12:27 pm
p.s. The latter option sounds most likely... Your bloody CIO should already know what a bad idea you not having privs on the prod box is. He shouldn't have to ask you. The fault might be partially yours, though. DBAs need to let people know what they do and why... even the trivial stuff. If they don't do it, things eventually fall on the floor and die.
--Jeff Moden
Change is inevitable... Change for the better is not.
July 4, 2016 at 4:52 pm
It could simply be a standard step in creating an exception to a security rule. Just document a few relevant specific tasks that require the level of sysadmin status to specific servers and services you'll need, and turn it in. If the 3rd party is following certain datacenter security certifications, this is a required step.
The bigger issue is awareness of how much of your position remains. What percentage of you workweek was spent on tasks moving to the 3rd party? If you're presently overworked, then this could be great news. Let other handle the backups, *you* handle testing them. If you will find yourself with lots of free time, however, then (as advised earlier in the thread) bust out that resume, add "assisted cloud migration of business critical app" to it, and see what else is out there.
-Eddie
Eddie Wuerch
MCM: SQL
July 5, 2016 at 6:38 am
Define "access to production"
Access to the underlying OS? I worked for 10 years without having that and everything was just fine (one or two minor snags, but nothing worth writing home about).
Access to the SQL Server instance in production? Well, actually, yeah. It's not about day-to-day operations though. Day-in, day-out, you shouldn't ever be on the production instance. Backups and maintenance should be automated and running. Deployments should go through a test process that can be automated so that you don't need to be connected physically to the instance. Any reports of slow running queries or issues along those lines should be investigated through monitoring tools without the need for a direct connection to production. Then there are the screaming emergencies. The database is offline. The server has failed over. Data loads have failed. The CEO can't run her favorite query because she's being blocked (or worse, was chosen as a deadlock victim). Now you need access to production. That's assuming you're still responsible in the event of an emergency. That's the easiest way I've been able to make it clear what I needed. If I'm responsible for emergencies, then I have to have a way to get there. If I can't get there, I can't be responsible for emergencies. It's that simple. From there, you right size your access, set up two different accounts (one for everything but production, one for only production), use the tooling available to ensure you can always tell when you're logged into production, have a change log, maybe even set up monitoring and auditing to record what you do when connected to production.
For me, it's all about that emergency.
"The credit belongs to the man who is actually in the arena, whose face is marred by dust and sweat and blood"
- Theodore Roosevelt
Author of:
SQL Server Execution Plans
SQL Server Query Performance Tuning
July 5, 2016 at 9:34 am
tezsb (7/4/2016)
Hello all,I am looking for help to justify my access to Production to my CIO. Sounds strange, doesnt it?
Well, it is, and I am at wits end here and feel like I am dealing with a bunch of fools :ermm:
long story short. We have an ISP provider coming in to take over the "infrastructure part of the company". But my role is not in scope (we will see). I am the only DBA in the company, and I work on both Production and non-production Databases.
This morning i was asked to document why I would need access to Production (both servers and databases). And short of "its my job to have access to production to keep it running", I am totally dumbfounded as to why they would even contemplate removing DBA access from Production instances.
Has anyone had this situation develop in their company where they have to explain why they need access to Production? To me it is like saying to a Pilot "you can fly, but you cant have a plane", you just cannot do your job.
Very frustrating....
First, you to ask your CIO to define what exactly your role is in relation to this specific server. Where I work there are production servers for which I don't have direct access. For example, there are database applications hosted in our data center for payment processing and financials that are supported by a 3rd party. It's blocked off by a firewall and only on rare occasions where they are in over their head with a technical issue would they temporarily grant staff DBA access.
"Do not seek to follow in the footsteps of the wise. Instead, seek what they sought." - Matsuo Basho
July 6, 2016 at 6:51 am
thanks to all for your replies, I appreciate the additional insight and view from an alternative perspective. Rightly or wrongly, I assumed the CIO knows what a DBA does - even though he used to be a developer.
So, I sat down and made a list of everything we DBA's do - literally everything from installing a sql server instance and maintaining patching, right down to creating a user, or database object. It was a tedious chore :doze:
Whilst I agree that deployments and things should be done in a test environment, and then deployed to Production via a script, I would still need access to production to deploy the script.
Anyway, I have had a frank conversation with the CIO and basically said that I cannot perform DBA responsibilities unless I am sysadmin on production.
It just beggars belief I had to justify it.
July 6, 2016 at 6:57 am
tezsb (7/6/2016)
thanks to all for your replies, I appreciate the additional insight and view from an alternative perspective. Rightly or wrongly, I assumed the CIO knows what a DBA does - even though he used to be a developer.So, I sat down and made a list of everything we DBA's do - literally everything from installing a sql server instance and maintaining patching, right down to creating a user, or database object. It was a tedious chore :doze:
Whilst I agree that deployments and things should be done in a test environment, and then deployed to Production via a script, I would still need access to production to deploy the script.
Anyway, I have had a frank conversation with the CIO and basically said that I cannot perform DBA responsibilities unless I am sysadmin on production.
It just beggars belief I had to justify it.
You may want to mention that without sysadmin access you cannot perform even routine taskis such as deployments, monitoring, or troubleshooting performance issues.
If the CIO calls you at 2am to report that the server is down, the only thing you could offer is your emotional support. :unsure:
"Do not seek to follow in the footsteps of the wise. Instead, seek what they sought." - Matsuo Basho
July 6, 2016 at 7:05 am
Eric M Russell (7/6/2016)
If the CIO calls you at 2am to report that the server is down, the only thing you could offer is your emotional support.
😀 I love this.
July 6, 2016 at 7:34 am
Beatrix Kiddo (7/6/2016)
Eric M Russell (7/6/2016)
If the CIO calls you at 2am to report that the server is down, the only thing you could offer is your emotional support.😀 I love this.
"Sorry, guys, but I'm not the admin on that server, so all I can offer here is a few words of advice and my emotional support."
Sometimes you'll be rewarded with a round of laughs... and sometimes they just won't get the joke. 🙂
"Do not seek to follow in the footsteps of the wise. Instead, seek what they sought." - Matsuo Basho
July 6, 2016 at 7:34 am
I love that too... Although I doubt I could even offer my emotional support... maybe my condolences
July 6, 2016 at 7:39 am
tezsb (7/6/2016)
I love that too... Although I doubt I could even offer my emotional support... maybe my condolences
Oh, I'd never earn MY paycheck offering emotional support. If that's what they expect from the DBA, then I'd have to polish up my resume and move on.
"Do not seek to follow in the footsteps of the wise. Instead, seek what they sought." - Matsuo Basho
July 6, 2016 at 8:20 am
tezsb (7/6/2016)
thanks to all for your replies, I appreciate the additional insight and view from an alternative perspective. Rightly or wrongly, I assumed the CIO knows what a DBA does - even though he used to be a developer.So, I sat down and made a list of everything we DBA's do - literally everything from installing a sql server instance and maintaining patching, right down to creating a user, or database object. It was a tedious chore :doze:
Whilst I agree that deployments and things should be done in a test environment, and then deployed to Production via a script, I would still need access to production to deploy the script.
Anyway, I have had a frank conversation with the CIO and basically said that I cannot perform DBA responsibilities unless I am sysadmin on production.
It just beggars belief I had to justify it.
Sorry... from that I can't tell so pardon a question with a possibly obvious answer. What's the end result of that conversation. Does he agree that you should have sysadmin privs on production or not?
--Jeff Moden
Change is inevitable... Change for the better is not.
July 6, 2016 at 8:30 am
Yes, the CIO finally agreed with me that I could not do my job without sysadmin... although for some strange reason the 3rd Party company might still insist on doing the backups "due to their contractual obligations to maintain the 5 9's". Well, as long as I can do an adhoc backup as required by business reasons, they can fill their boots.
Thanks again for all the great support on this frustrating topic 🙂
July 6, 2016 at 8:50 am
tezsb (7/6/2016)
Yes, the CIO finally agreed with me that I could not do my job without sysadmin... although for some strange reason the 3rd Party company might still insist on doing the backups "due to their contractual obligations to maintain the 5 9's". Well, as long as I can do an adhoc backup as required by business reasons, they can fill their boots.Thanks again for all the great support on this frustrating topic 🙂
They reckon they can maintain the 5 9's from offsite? I hope they've got good reaction times... They'll need a permanent connection to your environment in order to have a hope of keeping to that SLA.
Thomas Rushton
blog: https://thelonedba.wordpress.com
Viewing 15 posts - 1 through 15 (of 27 total)
You must be logged in to reply to this topic. Login to reply