October 6, 2011 at 10:44 pm
Comments posted to this topic are about the item The DR Test Schedule
October 7, 2011 at 2:20 am
The issue I have is: working for a small company, it's not so easy to lay hands on an unused server we can use to restore our data! Heck, even keeping spare mice and keyboards in stock seems to be slightly beyond the pale as far as the beancounters are concerned... 🙂
October 7, 2011 at 6:24 am
The issue I have is: working for a small company, it's not so easy to lay hands on an unused server we can use to restore our data! Heck, even keeping spare mice and keyboards in stock seems to be slightly beyond the pale as far as the beancounters are concerned...
Although it can be difficult to procure the items necessary for a good DR system, in your situation, it is important to document the repeated attempts. If things ever blow up, and I hope they don't, you can at least tell them "I told you so."
Thanks for the article,
Matt
October 7, 2011 at 6:30 am
While I don't have the responsibility for these servers, we do have images of them, as well as regular backups, and I suspect we could recover them fairly quickly. Maybe I should ask for a test someday soon….
Why not during PASS? 😀
If the junior DBAs at home can't pass a DR test while the "elite/MVPs" are out of town, then your DR plan is a failure. :hehe:
October 7, 2011 at 6:45 am
We have a DR drill every six months. All critical systems are tested every time. Others go through a rotation with the expectation that everything will be tested at least once every few years.
If you are short on money, consider using virtualized systems for your test. Remember, testing is not necessarily about the speed of the test system, but the ability to recover the data. In the event of a real disaster, the plan would include purchasing real hardware, but the recovery steps would be the same.
Our last test was a few days after the Irene and Lee dumped 8 inches of rain in two weeks. Our biggest concern: will the bridges be open so we can get to our test center.
October 7, 2011 at 6:54 am
I work for a relatively small company that invests heavily in technology. Our critical systems have full hardware redundancy with real time replication to our DR site. We ARE in hurricane country and we do fail over our systems to our DR site at least once a year. We have other contingencies in place also.
October 7, 2011 at 7:09 am
We do an offfsite DR test 2x per year.
In addition we are in a building with diesel generators and sufficient fuel to run for several weeks.
...
-- FORTRAN manual for Xerox Computers --
October 7, 2011 at 7:17 am
jay holovacs (10/7/2011)
We do an offfsite DR test 2x per year.In addition we are in a building with diesel generators and sufficient fuel to run for several weeks.
When you tried to run 24 / 7 on those generators, how many weeks did you last?
Do you check regularly that noone is "sniffing off" some of that fuel?
What happens if the fuel gets contaminated (I'd check here that everything is up to regulations, and highly unlikeky to be destroyed in the normal huge storms in your area).
October 7, 2011 at 7:25 am
Ninja's_RGR'us (10/7/2011)
jay holovacs (10/7/2011)
We do an offfsite DR test 2x per year.In addition we are in a building with diesel generators and sufficient fuel to run for several weeks.
When you tried to run 24 / 7 on those generators, how many weeks did you last?
Do you check regularly that noone is "sniffing off" some of that fuel?
What happens if the fuel gets contaminated (I'd check here that everything is up to regulations, and highly unlikeky to be destroyed in the normal huge storms in your area).
Good questions. We are located in a manufacturing plant and the generators are maintained by our plant engineers. They are regularly tested (and certified for enviro regs), actually kind of an overkill in capacity for just a data center, but it is a nice luxury. Two Caterpillar turbocharged 6 cylinder engines and one 12 cylinder Cat. (One of our company's lines is electric distribution switch gear so we have a nice setup).
They share the same fuel supply as can be used in heating units (we also have natural gas) so the fuel level is rotated. It's one big killer size steel tank at the back of the plant.
...
-- FORTRAN manual for Xerox Computers --
October 7, 2011 at 7:32 am
Ya my whole point is that untill and unless you try it, you just have no idea how much fuel "time" you really have.
I'd be assuming here that other system would draw power from that generator and that more might actually join in in a REAL dr situation.
It may sound like overkill but I'd ask to try and run the whole plant on a saturday strictly off those generators. Measure the fuel before & after and see if you forgot to plug anything that actually needs to run.
Also see how fast you can get delivery from the fuel company if you run out. Depending on the disaster, they might not be able to help quickly (if at all).
We had an icesotorm in Quebec a couple years ago and it took anywhere from 1 to 6 weeks to get electricity back (yes in winter in -20 conditions).
Fuel stations and convinience stores were all raided after day 1, not withstanding all the stores that had usefull equipement to sell.
October 7, 2011 at 7:47 am
We have run the plant on generators. We experience several power failures a year. The data center and emergency lighting, data center cooling etc. is the top priority for power allocation. Our data center UPS is the 'always on' type, street power feeds the batteries and the DC is run via inverter from the other end. The batteries (tested monthly) will run the DC for about 20 minutes however we never are down for more than about 3 seconds before the generators come on line.
As I remember the stock of fuel is about 26000 galons
...
-- FORTRAN manual for Xerox Computers --
October 7, 2011 at 8:09 am
Ninja's_RGR'us (10/7/2011)
jay holovacs (10/7/2011)
We do an offfsite DR test 2x per year.In addition we are in a building with diesel generators and sufficient fuel to run for several weeks.
When you tried to run 24 / 7 on those generators, how many weeks did you last?
Do you check regularly that noone is "sniffing off" some of that fuel?
What happens if the fuel gets contaminated (I'd check here that everything is up to regulations, and highly unlikeky to be destroyed in the normal huge storms in your area).
I once heard about a company that had generators with a couple of weeks of fuel capacity. The generators were tested every month. Then a real disaster happened, and it turned out that the monthly tests over a two-year period had drained the tanks down to where they had less than an hour of fuel available. Generators ran dry and the whole building lost power, and it took a while to figure out why.
So, yeah, there are aspects to managing generators beyond just "test them periodically" and that kind of thing.
- Gus "GSquared", RSVP, OODA, MAP, NMVP, FAQ, SAT, SQL, DNA, RNA, UOI, IOU, AM, PM, AD, BC, BCE, USA, UN, CF, ROFL, LOL, ETC
Property of The Thread
"Nobody knows the age of the human race, but everyone agrees it's old enough to know better." - Anon
October 7, 2011 at 8:51 am
jay holovacs (10/7/2011)
We have run the plant on generators. We experience several power failures a year. The data center and emergency lighting, data center cooling etc. is the top priority for power allocation. Our data center UPS is the 'always on' type, street power feeds the batteries and the DC is run via inverter from the other end. The batteries (tested monthly) will run the DC for about 20 minutes however we never are down for more than about 3 seconds before the generators come on line.As I remember the stock of fuel is about 26000 galons
Excellent. When I worked at the nuclear plant, we had the same thing. However since the admin building fed off the street, we'd accidently test the generators a few times a year when ice storms or something else caused lines to fail. We had power in the DC room, and thus coffee, but the rest of the building was done. The plant itself powers it's needs, but we also tested those generated (3 of time), one at a time every quarter, along with fuel levels, to see how they worked. Those things were amazing. Each one was literally the size of two large pickups end to end (though taller) with enough compressed air to jump start it in a few seconds.
October 7, 2011 at 9:13 am
Steve Jones - SSC Editor (10/7/2011)Those things were amazing. Each one was literally the size of two large pickups end to end (though taller) with enough compressed air to jump start it in a few seconds.
Bit off topic, I know, but I used to work for a company that made big electric motors and generators...now, standing right next to a 3000rpm, 40MW turbo generator as it spins up, THAT is amazing. Not to mention very, very loud! 😉
October 7, 2011 at 2:37 pm
Planned test - about once a year.
In some companies though we have had to use it once a quarter due to various things.
I read once upon a time that an awesome way to test your DR is through using it to setup QA, Dev and staging environments.
Jason...AKA CirqueDeSQLeil
_______________________________________________
I have given a name to my pain...MCM SQL Server, MVP
SQL RNNR
Posting Performance Based Questions - Gail Shaw[/url]
Learn Extended Events
Viewing 15 posts - 1 through 14 (of 14 total)
You must be logged in to reply to this topic. Login to reply