December 9, 2012 at 3:44 am
Comments posted to this topic are about the item Grace Under Pressure
December 10, 2012 at 7:06 am
[deleted]
Sigerson
"No pressure, no diamonds." - Thomas Carlyle
December 10, 2012 at 10:22 am
I agree most definitely with practice, practice,practice. However, with most Disaster Recovery scenarios I have had experience with in the last 20 years, the actual event never goes as planned in our dry runs, and they most definitely are different everytime around. Someone once told me "Don't panic!". Well... that is definitely easier said than done when you are in the middle of the lake up to your rear-end in alligators. I think that should be restated as "Don't do anything rash when you first feel that pang of panic." As far as managers raising their voice at you, I have a real good comeback for that and have used it in the past too. "Sir, do you have anything to contribute besides browbeating everyone here? If not, please leave the server room and let the experts think so we can analyze and address this issue for you as quick as possible, thank you.":-D
"Technology is a weird thing. It brings you great gifts with one hand, and it stabs you in the back with the other. ...:-D"
December 10, 2012 at 11:46 am
Steve, Honestly thank you for this. Very good piece.
I remember two disasters in the IT field over the years. One year back in the 70's the mainframe computer room caught fire and the emergency shutdown happened. This was a complete shutdown with a hard down and Halon filling the room. Two people were almost caught in the room when the emergency glass door closed to smother the fire. In the weeks that followed there were meetings, planning, and more meetings to replan and restart the center. Vendors were there 24x7 and people worked hard just to get the system back online. When management types would mouth off and start complaining they were taken into more meeting to work it out behind closed doors and the technicians were not there. The techs were insolated from the politics and screaming while they worked the problem and got the site operational as soon as they could. That worked.
The second disaster we when due to a single compromise in the engineering of the building. One water pipe entered or crossed the area where the main computer wiring traversed the building. In essence there was an unlimited supply of water traveling across the mainline channel which was lined with cement. Needless to say one day that water line broke and the water traveled down the channel directly back into the computer and it dumped out in the ceiling just above the racks of servers. Vendors and techs were on site within two hours and the entire site was back and operational in a few days. But again it was a very controlled project with those who had questions not given access to the techs.
In these cases cool heads and calm words bring out the desired result. Yelling and the blaming since it is counter productive needs to be removed from the area where productivity needs to optimized. This is a good case where if there is someone who really needs to yell should be instructed to go yell at the wall. 🙂
M.
Not all gray hairs are Dinosaurs!
December 10, 2012 at 12:31 pm
I remember a situation like that in the past - had a server issue that was being worked on by our tech guy and saw him being watched over by three managers, doing nothing except watching. (At least they weren't trying to be helpful at that point.) I ducked in, asked if I could do anything helpful, then expressed that I'd leave him to do the job at hand then. Never understood what they thought they'd get out of standing over his shoulder when they likely didn't even know what he was doing in the first place.
Shortly after that we had a better plan in place to test out a recovery plan. All backup tapes were sent there with a small number of servers that we needed to completely restore and get everything back online. We sent our main IT guy up to the center with pretty much nothing and he verified that he could restore everything to a functional state within the week. It was a bit painful, but it did ensure that we knew we could recover if something went critically wrong.
December 10, 2012 at 12:53 pm
Peter Schott (12/10/2012)
Never understood what they thought they'd get out of standing over his shoulder when they likely didn't even know what he was doing in the first place.
Peter, it's called micro-management, and many managers still actively practice it. Its a basic mistrust that your people need to be constantly supervised, even when they don't. It's a mind-set, and not an easily broken one either.:-D
"Technology is a weird thing. It brings you great gifts with one hand, and it stabs you in the back with the other. ...:-D"
December 10, 2012 at 2:19 pm
I've had managers who would come in and "supervise" even the smallest issues. I've also worked on projects where we would waste 1-2 hours a day in status update meetings.
Steve, I see you have very good taste in music... and album covers.
Tony
------------------------------------
Are you suggesting coconuts migrate?
December 10, 2012 at 2:41 pm
tabinsc (12/10/2012)
I've had managers who would come in and "supervise" even the smallest issues. I've also worked on projects where we would waste 1-2 hours a day in status update meetings.Steve, I see you have very good taste in music... and album covers.
Thanks! Been a Rush fan since the early 80s.
December 12, 2012 at 7:23 am
It is interesting that you are calling this Disaster Recovery Week. In my company, we strenuously avoid the term "disaster recovery". Instead, we use the term "business continuity"; we are not going to recover from a disaster, we are going to continue business. We have a colo site with SAN across multiple sites, replication, etc. We practice our BCP (Business Continuity Plan) regularly with the entire company running from the colo or "mirror site" for a full 24 hours.
December 12, 2012 at 11:11 am
I see this as two different items.
DR is recovering the systems back to a working state, or using new systems.
BC is the continuation of operation for your business. BC includes DR, but encompasses much more.
It's a valid point, that BC is really the goal.
Viewing 10 posts - 1 through 9 (of 9 total)
You must be logged in to reply to this topic. Login to reply