September 29, 2015 at 11:06 am
What do you do if your company doesn't truly understand what's a stake and decides that HA/DR is not important. We have a database that is the center of our universe and would cost 100 of millions of dollars if it were lost. We have nothing in place other than nightly backups. Company is not willing to fork out the money because they don't see the value in doing anything than just taking a nightly backup. What do you do at this point? Do you just go about your day even though you know if disaster strikes they will not own up to their decision but place the blame on you?
September 29, 2015 at 2:54 pm
Start with Brent Ozar's worksheet on HA/DR for a starting point of discussion.
http://www.brentozar.com/archive/2014/05/new-high-availability-planning-worksheet/
If you like your job, then the best thing you can do is try to work within the constraints of your budget. Certainly there are things you can do that will help mitigate risk. For example, move to full recovery mode and take frequent log backups. Then you're not out an entire day worth of data. (Assuming you have someplace to put the data.)
Another thing that stands out to me is the millions of dollars worth of damage being done. If your company is large enough to be making millions of dollars, are you under any sort of governance such as SOX? That's going to dictate that you have something more in place than nightly backups and good wishes. (We're going through a SOX compliance audit right now, so this is kind of on my mind ...)
Do you have a corporate risk officer you can talk to? Who owns the data that would be impacted if the database went down? If you have the database for finances (for example), then you talk to the VP of finance and let them know that their data is at risk. Try to find a high level sponsor for an actual HA or DR plan. I would like to think that someone up the food chain cares about their data.
If your job is just a paycheck, you're risk averse, and you think the company would treat you unfairly in the case of a disaster, then it's time to start looking for a new job. Ultimately, you have to do what's right for you.
This is where you have to decide if you're willing to try to work within the system you're in, or if you're going to escape while you can.
September 29, 2015 at 7:06 pm
I run into this all the time. The problem with DR specifically is it's not sexy. Execs love bragging about, "Now we can <fill in the blank> right from my Iphone and/or Android..." That's sexy to people who don't do. Not the same level of bragging that their data is backed up and ready to be quickly restored in a moment's notice.
In a business class I took in college (before I was ever had a job dealing with data) I had a professor who used to always say, "Show me the data!". This was in the context of: don't just make a suggestion and say, "its going to save us a ton of time or money". He suggested we have a spreadsheet ready with some real numbers to back up your suggestion. Especially if your data is governed somehow by SOX or HIPAA.
Nightly backup eh? 100's of millions if you lost the DB? That must mean that if the DB went down an hour before the nightly backup you would lose a bunch of loot. If you can quantify the dollar amount then compare that to the cost of a more comprehensive HA/DR strategy it becomes difficult to argue with. Worst case scenario you are covered.
I say argue your point - show them real data to back up your argument. It's not about just covering your butt, it's about making a difference.
"A leader takes people where they want to go. A great leader takes people where they don't necessarily want to go, but ought to be."
-- Rosalynn Carter
-- Itzik Ben-Gan 2001
September 29, 2015 at 8:01 pm
Thanks for the replies. Yes the money is very real. The reason it is so serious is because they only keep a couple weeks worth of backups which have never been tested. They are all stored on the same SAN where the live data currently lives. Problem is this was all setup by 3rd party vendors and my company never had a DBA before. I've been working on them to try to understand the severity but unfortunately none of them have ever worked in a real enterprise environment. They'd rather fight fires today than prevent them.
To give you a better idea of how it was before I arrived there were no backups being taken at all except of the VM. Now we are at least taking regular backups but I'm fighting for the time to work on it further. I spend most of the day being tasked with simple ETL projects that are highly visible and make people look good.
September 29, 2015 at 8:12 pm
The reason it is so serious is because they only keep a couple weeks worth of backups which have never been tested. They are all stored on the same SAN where the live data currently lives.
Yeah, that's just big bowl of wrong. If you have not done a test restore you can't really say that it's even backed up. Sounds like you've done your part. Sounds like you are right but I have learned that being right is not always enough. That's life. I've heard it said before and it's still true: have a good backup strategy or have a good resume (CV) ready to go at a moment's notice.
-- Itzik Ben-Gan 2001
September 29, 2015 at 10:42 pm
Just curious... how big is the database you're talking about?
--Jeff Moden
Change is inevitable... Change for the better is not.
Viewing 6 posts - 1 through 5 (of 5 total)
You must be logged in to reply to this topic. Login to reply