April 5, 2012 at 9:29 pm
Comments posted to this topic are about the item The On-call Demands
April 6, 2012 at 5:14 am
Being a DBA has changed my mind about IT. I am on call as probably most 24/7. I maintain 39 SQL servers. I am lucky however so far I have yet had an evening disturbed by a phone call, email that one of my servers has an issue. I am the only DBA. We have a lot of Exchange personnel and a couple of Enterprise Vault guys. It seems they are constantly being called in to fix this a or multiple problems after hours, don't get me wrong I have my fair share of issues, but I contribute a lot of my success to my father, he taught long ago to do a job right. I don't make a guess or let me see what this does. I want it done right before I turn a product out or leave for the day. So I feel lucky that even though after hours the clock ticks on down but because I try to be proactive while I am at work, my family time doesn't suffer. I also know eventually there are potential issues that will arise and the midnight hour will come and go where I will have to go in and fix this or that issue.
April 6, 2012 at 6:23 am
Depending on departmental churn rate and courtesy swaps, my on-call period falls on every 20-32 weeks. Where call frequency is concerned, we've managed to hire strong skillsets recently for 3rd shift so it's rare that we get more than 2 calls during the week, usually falling on the slot where there's no shift coverage (Sunday late PM). It's gotten a lot better, so no complaints.
April 6, 2012 at 6:31 am
Our 8 DBAs take turns, a week at a time. We have a mix of RDBMS species, so if we handle an after-hours call on a system we don't understand, we will bring another DBA into the conversation.
I'd say we handle around 3-5 after-hour calls a week. At this point, 95% of the issues are NOT MS-SQL 🙂
April 6, 2012 at 6:47 am
Grubb (4/6/2012)
I'd say we handle around 3-5 after-hour calls a week. At this point, 95% of the issues are NOT MS-SQL 🙂
That's always fun. I used to work on a mixed team (DB, AD, Exchange, etc) and most of my calls were non-SQL Server. Always interesting to learn something new, frustrating when it's under pressure.
April 6, 2012 at 7:16 am
At my current company call is about every 10 weeks. The only calls I've gotten are for batch processes that aren't automated and have regular problems for which there are standard fix scripts. Don't ask why the regular problems haven't been fixed :-D. That's only 5 times a week and at hours you are normally awake anyway.
At the last company I had call duties for we started with 9 in the IT department and ended up with 5 before my position was eliminated. We were on call for everything IT, from PC to network to Exchange to SQL Server. 99% of the calls were non-SQL related and had documented solutions. It still sucked at 3am. Averaged maybe 4-5 calls a week.
Jack Corbett
Consultant - Straight Path Solutions
Check out these links on how to get faster and more accurate answers:
Forum Etiquette: How to post data/code on a forum to get the best help
Need an Answer? Actually, No ... You Need a Question
April 6, 2012 at 7:45 am
2 DBA's, and we are on call every other month. The key to on-call is proactive automation and monitoring. Once you get that set up on all your db servers, you are emailed issues before they become major issues and you can deal with them during regular hours, thus preventing those late night calls like a T-Log that finally filled up a disk in the middle of the night. It doesn't prevent all of them mind you, but it does cut the volume down considerably. Also, having a well trained help-desk staff that can ferret out silly non-issues and simple connectivity issues before they get to you can help out considerably as well. If you have that all setup first then being on call is not such a nightmare. I have often found over the years through sheer experience that alot of after hours issues could have been caught during business hours if the checks and monitoring were already in place. There will always be issues in SQL Server that crop up quite suddenly and you must deal with them as they occur. However, I have found that 95% of most issues in SQL Server are as the result of accumulation. In other words they develop over time. This is where monitoring and automation come in. 😀
"Technology is a weird thing. It brings you great gifts with one hand, and it stabs you in the back with the other. ...:-D"
April 6, 2012 at 9:05 am
We have a team of 4 DBAs who are more or less on-call 24/7. As the team lead I usually get the first calls and will route them to the proper DBA as necessary. We average about 2-3 after-hours calls a week. About 40% of the time it is an actual database or SQL Server issue. That's what happens when you have 10 times as many developers as DBAs and the DEVELOPERS are the ones writing all stored procedures and creating/modifying tables and databases without DBA consultation or interaction.
Yes, you read that correctly. Welcome to my hell.
We also currently do not have ANY real monitoring and alerting in place except for what we have been able to slap together in between projects and emergencies. It is improving, however, since we will be bringing on one of the industry's leading SQL monitoring tools in the next few weeks. Then once we have everything dialed in we should have more time to prevent help the developers from destroying architect the databases.
April 6, 2012 at 9:28 am
bclyde-1080677 (4/6/2012)
... Then once we have everything dialed in we should have more time topreventhelp the developersfrom destroyingarchitect the databases.
:hehe:
April 6, 2012 at 12:04 pm
That's what happens when you have 10 times as many developers as DBAs and the DEVELOPERS are the ones writing all stored procedures and creating/modifying tables and databases without DBA consultation or interaction.
Yes, you read that correctly. Welcome to my hell.
I feel your pain. I can tell you though at my shop one of the first things I convinced my manager of is that we do not just hit F5 on any SQL code coming into our QA environment. We review all incoming SQL code for standards and performance metrics and if ANY developer code fails that review, then that code is returned to development for retooling. Period. There are no exceptions to this rule and we got managment buy in on it as well. Without management buy-in its useless. The code does not rollout unless the DBA group bless it, period. Some see this as a bottleneck, sure, but let a developer lock up an important production database with a badly coded stored procedure and the phones start lighting up, then managment sees the light real fast. 😀
"Technology is a weird thing. It brings you great gifts with one hand, and it stabs you in the back with the other. ...:-D"
April 6, 2012 at 12:54 pm
When I started at my current company I was the 3rd DBA so I was on call every third week. Before I joined the two DBAs would alternate, one would take day call the other would take night call and then switch back and forth every other week. Now we have 5 DBAs so I am on call every 5th week.
When we are on call it is 24/7. The night time calls are rare, maybe once a week or less on average, but an average is just that. We can go a week without night time calls and a week when the service center calls with a problem after midnight 3 or 4 times in the same week. (Sleep, what is sleep?) :hehe:
We strictly handle SQL Server support. No Access, no applications, just SQL Server. We have well over 100 instances of SQL Server to administrate spread throughout the state of Iowa.
Our manager wants whoever is on call to do just that - take care of issues as they happen through the day, freeing up the other DBAs to do their project work. It also frees the on call person from project work and allows them to do things that usually get pushed to the side such as documentation. 😀
This is the first time where I am part of a team of DBAs and I like it. There can be more stress, but it is a stress that I can deal with rather than at smaller companies where I would have to be DBA, Developer, Analyst, and have non SQL Server problems shoved in my direction and they want all problems solved at the same time. :w00t:
I just plan my schedule around the on-call week. No trips, no outings, nothing that would keep me from responding to a call within 20 minutes.
So it is a planned stress vs. a chaotic stress. 😎
-----------------
Larry
(What color is your database?)
April 6, 2012 at 1:01 pm
How funny, I got called this morning! I seem to get 2-3 calls a month. We have 200+ employees and a call center. I'm our software dev, so if something happens with the database it tends to affect my applications. So I get called on every glitch to either help troubleshoot the issue, fix it, or make the data right if our dialer didn't get loaded properly.
It never fails though, Mondays and Fridays are rip call days and always when on vacations.
April 6, 2012 at 1:05 pm
We strictly handle SQL Server support. No Access, no applications, just SQL Server. We have well over 100 instances of SQL Server to administrate spread throughout the state of Iowa.
I totally agree with most of what you say. However, the problem is with the above, is the people reporting the issue(error), as well as HelpDesk personnel on duty many times cannot always determine if the issue is database related, network related, or application(even Access) related. In many companies the Help Desk department has a constant turnover of people, so seasoned individuals that have the experience to detect the difference have often moved on or left. So, as a result, many times a DBA will get called for a wild goose chase anyway. Fortunately, in my company the Help desk people have been around for awhile and can usually ferret this out before the call is made. 😀
"Technology is a weird thing. It brings you great gifts with one hand, and it stabs you in the back with the other. ...:-D"
April 6, 2012 at 1:08 pm
TravisDBA (4/6/2012)
We strictly handle SQL Server support. No Access, no applications, just SQL Server. We have well over 100 instances of SQL Server to administrate spread throughout the state of Iowa.
I totally agree with most of what you say. However, the problem is with that, is the people reporting the issue(error), as well as HelpDesk personnel on duty many times cannot always determine if the issue is database related, network related, or application(even Access) related. In many companies the Help Desk department has a constant turnover of people, so seasoned individuals that have the experience to detect the difference have often moved on or left. So, as a result, many times a DBA will get called for a wild goose chase anyway. Fortunately, in my company the Help desk people have been around for awhile and can usually ferret this out before the call is made. 😀
My experience has been it is ALWAYS a database issue until proven otherwise beyond any reasonable doubt.
April 6, 2012 at 1:10 pm
Exactly my point, we get called on a lot of wild goose chases that had nothing to do with the database.:-D
"Technology is a weird thing. It brings you great gifts with one hand, and it stabs you in the back with the other. ...:-D"
Viewing 15 posts - 1 through 15 (of 26 total)
You must be logged in to reply to this topic. Login to reply