On call packages
-
Hi all,
My current company are drawing up an on call contract for our DBA team and another for our operations team. As a background the guy who used to do all the support has moved on and was under no contract so it is being drawn up from scratch.
This is my 3rd job and it's the first time and it's the first time a formal contract has been offered to me for "out of hours support" or "On Call"
What kind of thing should I expect to include or exclude? I realise that each company is different and has it's own systems so I am not asking which specific applications. I am looking for some kind of template that fellow DBAs follow.
Do you just get paid a flat rate?
Are you only paid if you receive a call/alert?
Do you get both?
What hours are you expected to work the following day if you have been up half the night?
That kind of thing.
Many thanks,
Rich
Hope this helps,
Rich[p]
[/p] -
The one thing I would absolutely demand is a backup. Make darn sure that you have a secondary oncall person. Otherwise, in theory, you can't go out & get groceries, let alone go to the movies or anything else.
After that, it really depends. I've worked at places where, because I'm salaried, being on call was considered part of the job. I've also been places where I was paid only upon the incident occuring (time and a half, minimum 2 hours, god I loved that job). My current position has a stipend it pays to the oncall people simply for the inconvenience of being on call, regardless of the number or length of incident.
Get what you can, but the best one in terms of money was the per incident pay. I jumped out of bed giggling when the alerts went off.
"The credit belongs to the man who is actually in the arena, whose face is marred by dust and sweat and blood"
- Theodore RooseveltAuthor of:
SQL Server Execution Plans
SQL Server Query Performance Tuning -
I tend to agree with what Grant wrote, but I've had a couple of different scenarios.
1. No on call pay. You get called, you work. That was not much fun and got abused at times.
2. Comp Time - We were required to give 8 hours of "overtime" a month, which could be working late, on call, etc. After 8 hours, we got comp time, meaning if I worked 9 hours overtime a month, I got 1 hour off the next month at some point. I think we had a cap of a week and hours rolled off after a few months, but that worked out ok. This worked out better for taxes.
3. Flat fee. We used to get $250 at one place, $300 at another per week of on-call.
4. Flat Fee + extras - At a large company, I got $300 a week for on-call, had to take I think 10 calls and then got $25-30 for each call after that. We didn't often run over 10 calls, but when you did it was a mixed bag: more money, big PIA.
Note that we also had to report how many calls we got and essentially "expense" the additional calls. That's not a lot of fun. I tend to prefer a flat rate.
Having a backup is nice, but more important than a backup is an escalation. If someone doesn't respond to a call, maybe in the shower, etc., then within 15 minutes a manager should be called and they should farm out the work to someone else. I'm not thrilled with backups because it puts two people on call when it's not usually needed. And then either they say you're the backup for no pay or the company appears to be wasting money.
Make sure that it's clear that you also respond when you're paged. You can't start working on something until you've let the person who paged you know that you're ready to work.
I'd get reasonable/realistic response times as well. I've had people say we had to respond in 5 minutes, which doesn't work well if you're showering or trying to wake up in the middle of the night. If you need that response time, keep someone on site. I think 15-20 minutes is pretty good.
-
Steve nailed it. I said backup, but what I meant was escalation. That's how we handle it.
I also agree on getting a realistic time frame. It takes five minutes for my laptop to boot up & VPN into the network. That doesn't count waking up, tripping on the dog, running into a door, cursing for five minutes because the kids moved your laptop, cursing for five minutes more because the kids left a toy in the office chair (not a big deal if it's one of daughters menagery of stuffed animals, but it can be QUITE painful if it's one of my sons implements of destruction)... Then log on to the server, wipe one eye enough to focus and read the error message... That's easily 15 minutes, no shower needed.
"The credit belongs to the man who is actually in the arena, whose face is marred by dust and sweat and blood"
- Theodore RooseveltAuthor of:
SQL Server Execution Plans
SQL Server Query Performance Tuning -
The secondary aspect to response time is how long (on average) it takes for the problems to initially be discovered. Our standards were "call back within 15, connected and working within 30", since we figured out that most calls we had received were on issues that were already at least an hour old (sometimes several hours). Hard to justify sending SWAT in when the bomb's already detonated. Not to say that it wasn't urgent, just trying to give enough time to be alert.
As to working and hours the following day - you make a lot of stupid mistakes when you're bleary eyed. It also makes you a LOT less effective at dealing with any issues that might come up the following night. We get a lot of flexibility with that, even with possibility of just working a few hours that day, from home.
Finally - WHAT qualifies to be "page-worthy" should be specifically outlined, or at least referenced (as an appendix, etc...) in this contract. Some issues can and should be addressed in the AM, and not in the 1-6AM phase. Nothing like getting paged (as a DBA) because someone's printer isn't working, because the networking guy didn't pick up the page.
----------------------------------------------------------------------------------
Your lack of planning does not constitute an emergency on my part...unless you're my manager...or a director and above...or a really loud-spoken end-user..All right - what was my emergency again? -
Matt has some good ideas. I'm not sure how easy this is to spell out in a contract, perhaps by system?
I've typically had to "train" people about what is worth paging and what isn't. Sometimes it takes some arguments with managers/directors over this. Course the bigger the pay for on call, the less stuff that gets paged out 🙂
-
Our appendix put systems (and apps on said systems) into a "tiered" system. Tier I - drop everything at all times. Tier II - urgent, Tier III "18 by 7" support, etc....
And yes - the appendix was a living document. But it was a good place to work out the rules with the community as far as "rules of the road". In our cases - no rules had meant "abuse whoever is on call with whatever issue you have", so even a partial document was a vast improvement. So - deal with the emergency, and if need be - adjust the call policy to fit.
----------------------------------------------------------------------------------
Your lack of planning does not constitute an emergency on my part...unless you're my manager...or a director and above...or a really loud-spoken end-user..All right - what was my emergency again? -
and just as important....train the callers to understand that at 3am, that on-call person is NOT as alert as they are. A good operational practice would be to issue a "This is a we have an issue call. We will give you 2 minutes to come to your senses/get awake and then ring you back."
The number of times I got called over the years, when absolutely dead to the world and expected to give a 100% accurate reponse within 30secs bears forgetting. It's a wonder there aren't more problems created in these moments than are solved by being on call.....:)
-
What hours are you expected to work the following day if you have been up half the night?
If you are in europe you get the european working time directive that says on average you can't work more than 48 hours a week - in the uk at least on call time is not seen as part of this unless you are actually working. It also says you are entitled to a miniumum amount of rest between working and being back in the office:
"every worker is entitled to a minimum daily rest period of 11 consecutive hours per 24-hour period" so if you work from 6pm till 8am - you should go home.
Personally I have been on call and called out in the middle of the night (12-7), as long as it doesn't take more than an hour i'll be in - anything over that then i'll have a lie in, if it took a few hours i'll have the day off.
I think that you need to be flexible, if there is a full on dcp operation and you have to rebuild at a new site then forget about any normal rules.
-
Hi,
Thanks for your replies everyone, they've been most helpful.
Yes my employers are based in England, I mentioned the 11hr rest rule but I was told that it was only an enforced regulation by one particular body and we weren't part of that body. I'm not convinced as I thought it was an EU law but I don't want to rock the boat until I have had a chance to look at it more thoroughly.
Currently it looks like there will be a flat rate with possibly some kind of compensation if you have to do a task that is x amount of time. But this hasn't been clarified yet.
Rich
Hope this helps,
Rich[p]
[/p] -
A couple of additional thoughts:
(1) Who pays for tools (e.g. home internet connection, cell phone, laptop/desktop)? Company rules regarding cell phone usage, home internet connections, etc. vary wildly and you shouldn't be paying out of pocket for tools that are required for the job. Once upon a time I got a $600 cell phone bill after a lengthy outage that involved a lot of international calling (site was in Canada and out of my calling plan)... company had a ceiling of $75/month on cell phone expenses, they eventually paid but it was a pain in the butt to get straightened out.
(2) Expenses while on call, if you have to physically go into the office are mileage & meals covered? Limits? Got challenged on 80 bucks for pizza for 6 geeks who'd been working a virus incident for 9 hours straight, in the office on a Sunday from about noon till well after 10 p.m.
(3) Authority to spend up to $X (no pounds on this darn keyboard) in the course of correcting an issue? I once had to carry almost $3K on my personal credit card for a couple of months after we had to call in a vendor on a Saturday night for a Sun box on which the support had been allowed to lapse... company eventually paid the expense report but it was a pain to get it approved as the amount "broke the rules", even with a verbal go ahead from my manager at the time. Obviously access to a company credit card or something would be a better answer but "what if?".
Joe
-
When I was on call, the company provided me the cell phone and laptop. The laptop had VPN so I could work from home when I got a call.
Usually there were two people on call, one was primary, one was secondary. If one did not answer the call or return the call in certain time, the operator would call the secondary.
In one company one primary on-call person did not answer his call 5 times, he got fired.
So make sure to ask your company the policy about on call.
You are lucky you get paid. Maybe I should move to England. When I was on call, I did not get paid and no comp time and I was expect to work my normal hour. One time I got a call at 2:00am and I fixed the problem and went back to sleep at about 3:30. Then I had to go to work at 8.
Again check with the on-call policy. If you get paid being on call, maybe they expect you to work your normal hour.
-
I've worked in a couple different environments in England.
Firstly, one where on I knew a bit of everything...So even though I was the DBA, I could get a call about citrix or a server down. I'd take a look and it if was beyond, you would just call the relevent person with more experience - It was a pretty horrible way to deal with support in my opinion. More times than not, I'd end up calling the citrix guys in the first few months.
Secondly, my previous job had 2 SQL Server DBA's and 2 Oracle DBA's. We just worked in a one week on, one week off format.
In terms of backups, the first role, everyone became the backup, although I was the only SQL Server backup as I was the only one with any real experience. The second role, we were expected to respond within 30 minutes. I was the backup if the support team could not contact the on call DBA and vice versa. It happened now and again for both of us but not too often so it wasn't really an issue.
As for pay, we had a flat day rate for Monday to Saturday and then double time on a Sunday and tripple time on a bank holiday and quadruple time when you was lumped with Christmas Day, Boxing Day and New Years Day! The policy was also that any call was chargable if it required you work for more than 30 minutes. We'd then get paid, based on our hourly salary. On top of that, we'd have patch windows (midnight to 2am or 5am to 7am) so that would be billable too.
It was ok and I made a good chunk of money working for this company but when you get maybe five or six calls through the night and weekends get taken up with calls it got too much for me and I left.
My current job is a kind of we call you, you work kind of role. There is no defined on call policy. I've been here 6 months and I've had maybe 4 calls. Of course, I get the odd weekend when doing an upgrade, but that's all planned.
I must admit, I prefer it now I am not on call. My basic salary is more, but overall I am on less without on the on call and my travelling costs are more but I'd not change it. When I lock my PC and head off, thats generally it for me ... I can go home and relax and play golf on a weekend without work calling me up and saying "we can see some database blocks....no, oh wait, they've gone, don't worry..." Not so bad on a weekend but at 4am...ggggrrrrrrrrrrrrrrrrr
-
Hi Richard,
Also UK based.
The on-call arrangements we currently have for 18:00 - 08:00 are a daily flat-rate which includes the 1st hour of call-out (overtime thereafter) with a higher rate on weekends and Bank Holidays. Response time is 'within 20 minutes' and theoretically employees should not live further than an hour's journey away from the office (in case disaster recovery is invoked and people need to be called in to panic instead of panicking more quietly at home where they can probably be more effective). Although the EU 11 hour rule is not strictly applied, a commonsense approach dictates that if someone's been up half of the night resolving issues, their presence bright-eyed and bushy-tailed at 8:00 will not be expected (usually resolved by coming in 'late' or working remotely, staffing levels permitting).
Ops have an extensive list of 'actions to be taken' depending upon the nature of the problem so call-outs are largely devoid of trivia and therefore less frequent. One point on this though is that our dev and test systems sometimes get upgraded to production status as release dates approach and seem to stay that way for longer than intended - in other words try to ensure that your on-call is in response to production problems, not just because someone kicked off a new process on a dev server before going home and it's filled tempdb by 2 a.m.
The other thing is to make sure that enough people with the requisite skills are on the on-call rota to cover holidays, sickness etc (and to have an idea of the likelihood of being called). I mention this because you may need to share the load in extremis (I was once called as I approached my front-door on Friday evening and was still resolving issues at midnight on Sunday evening as the other DBA was on leave and, yes, I did have to go into work on Monday morning).
Escalation managers are useful but if you're the only resource available you can end up with more grief as the 'problem' passes up the chain and then back down again with more levels of management querying what you're doing and why aren't you doing it quicker!
Too long in the tooth and cynical (moi?) these days to sign up to sanity-threating conditions of service. On-call can be fiscally beneficial if you're rarely called but not worth the rub if it makes your health or home life suffer too much.
Take care.
-
I've never had a contract, but these are issues I have learned to discuss with my boss before I take a job:
If I work after midnight, I come in late (time being defined as "when I wake up"), as long as the issue got resolved.
Issues at the office:
If you have a batch operator/night time operators. Make friends with them. Work with them. Empower them to fix problems that there are no risks in them fixing. Massive reduction in calls from this. This of course depends on how much you can trust the person, but that will build up over time.
Viewing 15 posts - 1 through 15 (of 17 total)
You must be logged in to reply to this topic. Login to reply