Disaster in the Real World - Hurricane

  • Comments posted to this topic are about the content posted at http://www.sqlservercentral.com/columnists/

  • Thanks for the article. I found it informative as I am not used to Hurricane's in South Africa. We don't have Hurricanes's, Tornado's and earthquakes!

     


    Andy.

  • Andy: Nice article.  I wish I'd had time to write a full article, our experiences were similar in some ways, different in others.

    We provide a semi-public SQL Server for our customers to trade info with each other, as well as web sites and email services for them.  Those servers need to keep running.  I live on the north east side of Orlando, and had quite a bit of tree damage as well as minor house damage (ripped out soffits, total destruction of pool screen, tree falling across the power line to our private well - minor stuff).

    Needless to say, we lost power Friday around 8:30 and didn't get it back until the following Friday night.  Judging from the number of very large trees ripped out by the roots or snapped in half on our property we got a pretty good wack.

    We do have a different setup in the server room though.  We run all the servers off DC, all the time, including a small air conditioner.  We use a couple of old telecom battery chargers to charge a bank of batteries, and a set of 48VDC to 120VAC inverters (Exeltech MX series - which I love) to keep the servers and AC running.  We get about 7 hours run time on the batteries.  We also use a 5KW gas generator as a backup power supply to recharge.  We worked it to near death this week, but never had to shut down the servers.

    Our next move will be to get a "whole house" style generator that can kick on and keep the whole office running, not just the server room.

    Glad you're okay Andy, come take a tour and get a free lunch sometime.


    Student of SQL and Golf, Master of Neither

  • Glad you all survived.

    I'm in SW Ohio and the results sound sort of like the combo of a blizzard and ice-storm.

    My personal rule of thumb is that you need to set yourself up for three days without outside support. My example is 3 days of food & water, and in my general case heating supplies. This is assuming your house is mostly intact. If we lose power in the winter, the fridge goes into coolers set outside.

    At work, they still haven't nailed it down like that. But I'm just a peon in the DR plan.



    ----------------
    Jim P.

    A little bit of this and a little byte of that can cause bloatware.

  • It would be interesting to know your command and control structure during a disaster. With cell phones not working, were any high-level members of management not able to make decisions? When do lower-level managers make the decisions to bypass the higher levels because of lack of communication? It sounds like there was a documented plan and everyone knew the steps and contingencies, but who was making the decisions? Does IT control the business during a disaster or is the business still in control of operations?

  • Last week I participated in a State wide disaster exercise in a local "disaster" drill playing a plane crash victim. Part of the drill required that we not bring cell phones (or anything else for that matter) so we would not be distracted or more probable, loose them in all that was going to happen to us. I don't live in hurricane country but do live in an earthquake and volcano prone area. Being without power in the winter (below zero temperatures) for 3 days prompted me to get a generator long ago.

    My plan was to leave the cell in my vehicle and check on it, if we got a break. No break came for the entire day so no chance to check. You can guess what happened. The main server and network crashed 3 times that day. It had been running smoothly for months without problem. I am a one person IT shop. Part of my backup plan relied on another individual who had just left the company a week or so before and a replacement had yet to be determined. That persons job is to follow a set of procedures of "if this - do this" instructions. Fortunately, the automatic recovery worked each time and the day went on. Point is, disaster strikes at anytime not just in bad weather conditions (yesterday we had 80 mph winds blow through, no problems) and one must keep that disaster plan up to date. That reminds me, I'm still waiting for the buildings backup generator to be installed.

Viewing 6 posts - 1 through 5 (of 5 total)

You must be logged in to reply to this topic. Login to reply