Disclaimer: From
the soapbox does not necessarily represent the views of SQLServerCentral.com,
Microsoft, or the author himself.
So you have full backups,
differential backups and transaction log backups enabled and functioning for
your production databases. You have
documented procedures for restoring databases, and you perform quarterly tests
of the process. Is this disaster
recovery for SQL Server?
No.
Disaster recovery is NOT the
equivalent of a backup and restore plan; it encompasses much more than that.
To be sure, this is an integral part of a DR plan, but it is still only a
piece of that plan. Here are some
common beliefs I've encountered; in fact, I've held some of them myself at one
point or another in my career:
"Our server staff backs
up all of the databases with product X -- I don't need to worry about it". I really hope there's no full-time DBA that would say this.
I've run into this with “pseudo”-DBAs -- developers or systems staff
that have the responsibility thrust upon them.
While a centralized backup methodology in a large environment makes life
much, much easier, you need to be certain that whoever manages that backup
solution understands the specifics of how to properly backup a SQL Server
database. More than once I’ve
seen a DBA-less company use an open-file backup solution on a database; it’s
the easiest way to get introduced to dealing with a “suspect” database.
Oh, it's so much fun.
"As long as my databases
are backed up, my responsibility as a DBA ends".
At one point earlier in my career, I actually believed this. At the time
I was working with a company where IT was organized in "silos" -- all
of the DBAs were in one group, the systems people in another, etc.
This made cross team interaction difficult, both from a communication and
logistical standpoint to a cultural one. Each
group had very clearly defined areas of responsibility, and we all clung to them
to ensure that we didn't get blamed for an issue that beyond our control.
I now fervently believe that it is irresponsible for a DBA to simply stop
with the backups of databases. It
doesn't do much good to have a solid database backup plan if there's no plan or
process for recovering the server itself.
"As a DBA, I know the best
way to set up a backup scheme for my databases".
Of course, the DBA knows the best technical way to set up backups. The scheme, however, should always be dictated by
business units. There are three
questions that always need to be asked of any business unit:
1) What is an acceptable amount
of data loss
2) What is an acceptable amount
of downtime
3) How long do backups need to
be retained
Initially, the answers will
almost always be 1) "None", 2) "None", 3)
"Forever". When that
happens, simply explain the costs of co-location, data archival services, SAN
implementation, clustering, etc. At
that point you'll get a reasonable answer. Your legal department should also be
involved, because there may be legal ramifications of data loss unknown to the
business unit. From there you can
build a technical backup scheme -- how often you backup up, do you need to use
differentials, how long do you retain transaction logs, etc.
This plan and it’s ramifications should then be reviewed with the
business unit, legal, etc. Having them sign off on it is also an advisable thing. 🙂
"Disaster recovery is the
responsibility of IT". While IT plays an integral role in the development
of the plan, it is ultimately the responsibility of the business itself.
Executives and management need to understand the impact of each disaster
scenario, as well as what's required to get up and running again.
Business units must have some sort of contingency to continue operating
without access to an application. Facilities staff must be ready to correct
power, HVAC and environment problems (such as water leaking into the server room
-- has anyone else had the joy of dealing with a soggy server?).
Again, I believe it is irresponsible for a DBA not to raise awareness of
the company's entire responsibility towards disaster recovery.
"I'm just a DBA -- I can't
make a company-wide disaster recovery plan happen".
Ah, the sweet smell of despair -- I've been there myself.
It is true that you can't make a company-wide DR plan, but you can
make it happen. Talk to your
manager about concerns you have. Talk
to business units and their managers. Pound
on the CEO's door as if you're storming a castle. As long as you are tactful and
professional in your approach to driving change (avoid wandering around the
halls beating a drum and shouting "We're doomed!!" – take it from
experience, it doesn’t go over too well), chances are someone will eventually
listen, and you're efforts will be appreciated.
“I have a full backup scheme
in place – I can recover to any point in time.
I have redundant hardware, off-site backups, transaction logging, etc.
We could lose this entire facility in an earthquake, and I can still
restore the database to another server up to the latest committed transaction.
The database can be recovered in the event of any disaster.”
Any disaster? Are you sure?
What about the data entry clerk who’s been inconsistently entering
incorrect data? Sure, you can recover to a point in time, but when is that point
in time? And how disruptive will it be to the use of the application to attempt
this type of restore? User error is the most frustrating and difficult disaster
scenario to deal with. When the CEO
is standing in front of you demanding that you “fix it”, you still have to
deal with it, whoever is responsible.
“Our company understands the
true needs of a disaster recovery process – we just don’t have the time or
resources to undertake a project of this size.”
Creating a DR plan doesn’t have to be a major project – in fact,
it’s less likely to be successful or useful if it is undertaken as a single,
all-encompassing project. Take things in small steps and create the DR plan as you go.
You could start by simply writing down the names and phone numbers of
anyone that needs to be contact, from internal staff to outside vendors (power,
plumbing, HVAC, etc). Document restore procedures.
Store backup information off-site. The
point is, take it slow. Disaster recovery planning should be viewed as an
ongoing process, not a single task to be completed.
If any of these scenarios sound
familiar, you could be heading for trouble.
At least that’s what the view looks like from up on the soapbox.
Am I wrong?
Am I right? Am I just plain
crazy? Just because the rant is
over, it doesn’t mean the debate is resolved.