As you may have read in the newsletter, I'm lucky enough to live in the
Orlando area, right in the middle of the path Hurricane Charley followed. What
follows are some of the problems I've experienced both personally and
professionally over the last few days. Hindsight is 20-20 they say, but I'll
throw some of those comments in too.
My first observation is that most people in Florida haven't been through a
hurricane and as much as you might try to explain, it's not the same. I saw a
comment in the paper from a man who had been playing golf at 6 pm on Fri and had
been telling friends that he had played in far worse weather. Quite a different
story three hours later. I spent two months in Miami after Hurricane Andrew -
the pictures just don't do it justice. Catastrophic is a pretty good word.
Anyway, I think before you are allowed to live in Florida you should have to
tour a replica of a post hurricane city. Rant complete.
We were watching the storm at work and even as late as Thursday at noon we
were planning business as usual on Friday. At that point we would have been
catching the outer band of the storm, probably 30-50 mph winds, heavy rain. I
decided Thursday afternoon to have my team work at home on Friday, playing it
safe, but everyone else was scheduled to be in the office. Late Thursday we
decided that Friday would be a half day. About the same time we found out all
schools would be closed Friday. In hindsight (as promised), I think we would
have done better by our people if we had closed or went to minimal staffing.
It's tough, you balance losing billable hours against not giving yourself enough
margin. As we wrapped on Friday we were still expecting business as usual on
Monday.
I spent Friday afternoon prepping for the storm. Making sure we had
batteries, water, etc, plus securing all the loose stuff outside - Florida homes
tend to have lots of chairs, pool floats, and other stuff that becomes deadly in
full hurricane winds. At 2 pm the power went out. Just very light rain at that
point, no wind. The power company said it was a fuse out on a pole but that they
were deferring all fixes until after the storm passed. Mind you that the storm
wasn't due to arrive for another 7 hours. Summer in Florida without AC is
miserable. I sent my wife and baby girl to a local hotel, sat down next to a
window to catch up on some geek reading.
About 8 pm the power went out at the hotel! Ah well. At least the room was
very cool and the heat of the day was gone, too late to try moving them with the
storm an hour away. Just about 9 pm the storm arrived. Fairly serious wind, but
in my area I don't think it exceeded gusts of 50-60 mph. The whole thing was
over in an hour or less. No damage to the house, hotel fine, sit back and wait
for morning.
Sat morning I survey the area. Couple minor tree limbs down in my yard, tons
of leaves and small branches, nothing serious at all. Pretty similar in the rest
of the neighborhood. My neighbor across the street has his generator running -
must be nice! We've been house hunting and had planned to acquire a generator
then, so I had put off buying one because I wanted one permanently installed,
didn't make sense to do for a house I was going to sell and I hated to buy a
portable one that I wasn't going to need later. Hmmm...maybe not the best
decision. If having a generator installed at your home seems like overkill,
trying going without ice or AC or anything else electric for a few days. It
varies by area, but we've averaged 2-3 power outages of 12 hours or more where I
live per year for the past few years. Is there a lesson here? Perhaps, but that
20-20 thing...
Over the past couple years we've seen people become more and more reliant on
cell phones, in many cases opting to stop paying for a traditional land line.
Others have turned to VOIP services such as
Vonage that tend to need a broadband connection, usually cable to work. At 8
am Sat morning impossible to get through to anyone on a cell phone. Even today I
can't reach my mom across town on her cell phone, though in most other areas it
was ok. Local authorities were asking everyone to not use their cell phone as
they needed all the available bandwidth to help with the recovery efforts. We
use Nextel at work, phone calls would not go through during much of Saturday,
but the radio part seemed to work better. I don't know, but I wonder if cell
carriers don't have a way to prioritize emergency traffic. Landline worked the
entire time.
Mid morning Saturday I move wife and kids to my mom's, on the way back our
network dude calls. We had switched critical systems over to run on the building
generator, but the generator doesn't have enough power to run the AC - our DR
plan calls for us to rent portable AC units if power will be out for more than
few hours. He had the units, but somehow the last time the generator wiring was
changed it was done in such a way that the elevators had no power! It took four
of us to get two of these things up to the third floor. Two pulling, one
pushing, one holding a flashlight because there was no stairwell lighting
either. Brand new they cost about $2k - maybe we should have just bought them?
Even with only mission critical systems running, the server room was up to 99
degrees when we got the coolers plugged in. Just can't get the heat out even
with a decent sized fan. Four hours later it had dropped to a hot but manageable
92 degrees.
Driving is crazy. Enough major lights are out to make left hand turns a
danger, everyone seems to have forgotten the rule that if the lights are out to
treat as a four way stop.
Sat night, still no power. Not fun. I've got two large dogs and a cat, no
room a my mom's, I've got to stay at home.
Sun morning I stand in line at Lowe's to get a generator. They ran a first
class operation. People who had been there to get one on Sat had been given a
ticket to return for a truckload due in Sun morning. Sun morning they had two
lines, one for people who had tickets, one for people who did not. Officer
Morris from the Sheriff's office was on hand to keep order (I mention this even
though he may never see it, but he was polite, patient, helpful, cheery -
everything you could ask for in a civil servant). Several times the store
manager came out to make sure everyone knew what was going on. Once they had
everything organized, they had the people with tickets entire single file, then
the rest of us followed behind. We passed by stacks of 5 gal cans in case we
needed them, then went all the way to the back of the store. I'd say it probably
took an hour for them to hand out 300 generators, took another half an hour to
check out and head back home. Lowe's did a first class job, couldn't ask for
better. Not all businesses were as well managed or responsive, or understood how
to communicate to help manage the stress that comes with times like these.
I head up the road for gas, the station has power but no gas. Go home, drop
off the generator, go back out to get gas. Find gas, come back, generator starts
on the first try. Always a nice thing. The generator holds 5 gals and is
supposed to run for 11 hours, so I head back to Lowe's to get another couple
cans and then get more fuel. No 5 gal cans at Lowes, sold out. I grab a couple 2
gal cans. Spend 45 minutes looking for gas. It's always something isn't it? The
stations that had power sold out quickly, but there were tons of stations with
plenty of gas, no power.
Back at home, I throw out all the food from the refrigerator. What can you
do?
As I'm driving around Sun most of the major stop lights are working, either
on main power or a generator, a few officers directing traffic where there is no
power. There are a few fast food places open. Sun night we go out to dinner at a
local restaurant. At the same time we are hearing about the extent of the
damage, ice and fuel shortages, that the power might be out for a full week.
Power customers starting to get irritable, lot's of situations where the house
across the way had power but they didn't. Power companies prioritize their
efforts to get banks, hospitals, police, fire, other important things running,
residential customers last on the list. I don't doubt that they were working as
hard as they could, where they weren't doing a good job was conveying to the
customer base where they were making headway, what the challenges were, a better
guess at a timeline for specific areas. To me, this is one of those times where
you give out as much information as you can. People may not like what they hear,
but they should be able to see some progress and understand the complications. I
suspect the power companies don't like to get into that kind of detail for fear
that they will then spend more time defending those decisions (or defending why
it's broken to start with). People don't like unknowns, better to let them know.
Sun at noon our plan is to have our main office closed (no power), our remote
office open. We'll have everyone report there, figure out who we can use, send
the rest back home. Not my favorite plan, I thought we'd have done better by
everyone to just stay closed Mon. At noon there were still streets blocked in
some areas, a fuel shortage, schools closed on Monday.
Sun at 4 pm we get main power back at the office. We change our plan to have
everyone to report for work as usual.
Mon six of eight people on my team are able to make it in, not bad. Life is
pretty normal at the office. No damage to the building, no data loss, life is
pretty good. No where to go for lunch, but that's ok. Lot's and lot's of our
employees have no power, are struggling to find ice and fuel. By Mon night they
have made some progress restoring power, but it's going to be Saturday before
Central Florida is 100% restored, and Thurs/Fri for large pockets of residential
customers. Orange County (Orlando) next door to us announces that schools will
not resume until the following Tues. My wife finds a gas station, brings over 2
5 gal cans, enough to run the refrigerator off and on long enough for cold water
and ice for a couple days.
Tues (today) sees some progress with power restoration, just not my power!
Still forecasting Thurs or Sat depending on who you listen to. Local schools set
to resume Fri here in Seminole County. Work is normal, more places open.
We were lucky in my area, not much damage. Other areas not far away fared far
worse. The only real lesson I see here is that good communication - and the
ability to communicate - are huge parts of managing a disaster. I imagine a lot
of you can relate to the last time you had a major incident at work, how many
people stuck their head in the door saying "how much longer?". I talked to Steve
about this some and he had some nice bits of info about how he has seen
companies prepare for and handle incidents, hopefully he will find time to write
some of that up soon. Beyond that, I think a great DR plan is one where you
really war game scenarios, just to see what you might could do. If we had been
on the 20th floor that whole no power to the elevator thing would have really
hurt!