DBA vs Sys Admin
-
I don't know if this topic has already been beaten to death, but maybe someone out there has some answers.
Here’s my situation…I was recently hired on as a consultant with a company who recently experienced a major disaster during which they found they could only recover back to the previous week. Why? Well, first of all, there was no DBA. Second of all, the SAN administrator and Sys admin were in charge of backups and recovery. Their backup plan was to backup the .mdf files directly to tape each night and hope for the best. No transaction log backups, and if the full nightly failed I guess they figured there would be another chance tomorrow night.
At this point (after the disaster) they understand they need for full backups plus transaction log backups throughout the day. I’ve helped identify the critical systems and set schedules for backups. The problem is that they insist on doing the backups directly to tape even though there is enough disk space to accommodate disk to disk(plus more has been ordered). My recommendation to them has been to allow me to schedule the nightly backups through sql as well as the tlogs, then they can backup to tape after midnight, but I’m not getting through! The net backup through Veritas that they are using doesn’t seem very dependable. I check the error logs each morning and can see at least 1-2 database backups failed during the night daily. Their answer to this is sometimes the backups run over the time allowed in the policies and will abort and they’ll look into it. I would like to have the .bak file on disk for easier restorability and my own peace of mind(as opposed to the mdf files), but it’s like talking to a wall! If a database needs to be restored they insist they can do through the backup software. They don’t see the need to change their methods. The typical reply is that tape backups are reliable enough.
I’ve been offered a full time position with this company and would really love to accept, but how can pretend to be a dba if I can’t even guarantee the backups and restoring of the data? Has anyone else had this issue?
Anita
Thanks!
-
I would insist that you are given the right to make these decisions as as a prerequisite of accepting the position. When I started at a new company, two of the major production servers that I was tasked with looking after where being backed up hot to tape via Veritas because of disk constraints. I complained, had my concerns noted in emails to all concerned.
One of the boxes bit the dust shortly after I arrived and myself and the dev team spent the weekend getting the rebuilt box back in service. What I found out was a few of the databases where not being backed up, the veritas job runs off a script and newer databases had not been included in the script. The DBA team had nothing to do with the veritas job, and although we were able to restore using backups that had been used for dev and stage we lost days of data.
I now have a job that polls all my servers to check that all the databases are being backed up, and we back everything to disk before it goes to tape, and suffice to say the guy who was in charge of the veritas backups is no longer with the company.
-
From a DBA's perspective, first and foremost you need to guarantee the databases are successfully being backed up. Once you have a successful disk backup the file can be shunted anywhere, another storage location, tape, etc...
I've been bitten two previous roles by backup agents and have successfully removed them in the half-dozen roles I've undertaken since then.
If disk space is a problem the cheapest option these days is to buy more space. If you don't want to take the hardware route, take a look at using something like Litespeed or Redgate's SQL Backup. I've used both products and have a greater degree of comfort with them than either of the backup agents that I've seen.
In all cases the action of performing the initial backup of the databases should be in the hands of the DBA. The System/Network Admin should only be involved in the historical storage of the backup files.
--------------------
Colt 45 - the original point and click interface
-
Without a doubt a pre-requisite for accepting a permanent position with this company is that you will be in charge of database backups and the Sysadmin should be responsible for tape backups, 2 separate tasks. I would be wary of taking a permanent position at a company that has experienced the problems that you note and they still refuse to listen to a consultant that was brought in to fix the issue.
Jack Corbett
Consultant - Straight Path Solutions
Check out these links on how to get faster and more accurate answers:
Forum Etiquette: How to post data/code on a forum to get the best help
Need an Answer? Actually, No ... You Need a Question -
Another point that may be worth bringing up is the speed of recovery. Recovery from disk will almost always be faster than recovery from tape. That would allow you to recover quickly if there is an emergency. Another argument to be made would be the redundancy of the backups. What if a tape is bad? You can't recover from that tape. What if just one of those TLogs on the tape is bad? Can't recover past that point.
What happens if you add a new database or if a dev adds a new database? The software I've seen for tapes requires that you add that database to the job. It won't pick it up automatically. If someone forgets to mention it for weeks - you have nothing to work with during that time. From what I've seen, SQL is pretty easy to configure so that all databases are backed up.
We had the window at one point to allow backing up to Tape via some open file manager software, Tape via an Agent for SQL Server, and File through SQL (which was also backed up to tape). I think we eventually dropped the open file manager bit, but still had backups to tape and file.
-
I've brought up every single of the great points you guys did in the last two weekes, but I think this sys admin just doesn't want the backups taken away from him. I even told him that from now on it would be on me if we can't recover...you would think that would make him feel better, not having that responsibility, but guess not. I talked with the director who agrees with me and hopefully we can solve this issue without the hurt feelings and fighting. Thanks for all the replies!
Thanks!
-
Can you schedule the backups through SQL at a time they aren't running their tape backups?
Easy way to ensure that you get a SQL level backup regardless of what's going on, especially if you have the backing of the director to do that. After all, you're just going to do it as a test, right? 😉
-
Ultimately, though, it's not the system administrator's call (speaking with my system administrator's hat on). Data retention and recovery requirements are business requirements. They are driven by the users with the advice of the technology staff. I know, I know, pie-in-the-sky ideals.... However, if there is one manager over both the DBA position and the system administrator, it's ultimately that manager's call.
It sounds like you've got your ducks in a row as far as what the issues are. The catch now is to translate that into business cost... if you can show the potential downtime and data loss with the information you've already collected, you can share that with management. Ultimately it should be their decision on whether or not to back you up or not. If they don't, even after you can show business loss, you know all you need to know then, don't you?
K. Brian Kelley
@kbriankelley -
I would, but I need the sys admin to give me a volume on the SAN to do so! I'm stuck for now, but I think with the director talking to him we SHOULD get this resolved, but I'm wondering at what cost? Will my life here be miserable now?
Thanks!
-
Well, if the director has tact, then your life should be decent. After all, it's a business decision and if the backups aren't working successfully now (and backing up MDF's directly?) then the problem needs to be addressed. Hopefully he will approach this in a way that says - let's test this and see how it works. If it doesn't work, you've got the tapes to fall back on. If it does work, you have redundancy. Besides, the director should be the one dealing with the sys admin so you should be out of the line of fire. The SysAdmin still has to ensure that those files get backed up to tape - just a change in the job there.
Any way you can stick out the contract a little more to see how things play out before accepting a perm position? If it gets really ugly you can move on without too many problems. If not, then you can consider whether this is the place you want to work on a perm basis.
Viewing 10 posts - 1 through 9 (of 9 total)
You must be logged in to reply to this topic. Login to reply