Stop, Don't Panic

Question

Stop, Don't Panic

Steve Jones - SSC Editor

SSC Guru

Points: 736494
More actions
March 15, 2010 at 9:15 pm

#137893

Comments posted to this topic are about the item Stop, Don't Panic

Viewing 15 posts - 1 through 15 (of 30 total)

You must be logged in to reply to this topic. Login to reply

John Summers SSC Enthusiast Points: 151 More actions · Answer 1

Very much been there, done that. Since I work in a manufacturing environment (7X24). I can add my own rule to "don't panic" - sleep on it. Between midnight and 6AM, when production wakes you up. Your monitor processes are e-mailing, texting, sounding the sirens, you've got a radical idea on how to fix the whole mess... stop, just patch enough to limp through until morning, get some sleep, go in and go over your ideas with somebody more sane (or at least more rested). I have more than once realized that what I would have done at 3AM, would have only added fuel to the fire.

Tired people make really stupid decisions. Be aware of this

John Langston SSCommitted Points: 1977 More actions · Answer 2

Have a plan, at least a generic outline of what you will do under given circumstances. It's a subset of what management should have in place for their business continuity plan. Management has one right? So should you.

Consider in advance the steps you would run through, the checks you would make and why you would make them. The plan may help yourself and others really understand the purpose of the data housed on server A and who consumes it. Having a list of data consumers is the starting point for that notification list of business people who need to be in the loop when disaster strikes.

GSquared SSC Guru Points: 260824 More actions · Answer 3

Yep.

Or, to put more explictly: I agree with the editorial. Far too often, I've seen people react to stress by doing stupid things.

On the other hand, after you've had to reach inside someone's body to clamp a bleeding artery shut with your fingers, nothing IT-related seems all that panic-inducing. That may have something to do with my ability to remain calm in emergencies.

- Gus "GSquared", RSVP, OODA, MAP, NMVP, FAQ, SAT, SQL, DNA, RNA, UOI, IOU, AM, PM, AD, BC, BCE, USA, UN, CF, ROFL, LOL, ETC
Property of The Thread

"Nobody knows the age of the human race, but everyone agrees it's old enough to know better." - Anon

Steve Jones - SSC Editor SSC Guru Points: 736494 More actions · Answer 4

You hope management has thought things through, but often I've found they haven't. It's only the people that have been through things that tend to have plans in place, or a general idea ahead of time.

Pushing back on management to make sure things are fixed properly is hard, but hopefully you feel empowered enough to do it.

sjsubscribe SSCrazy Points: 2080 More actions · Answer 5

Reminds me of the line, make haste and repent at leisure.

Ross McMicken SSCarpal Tunnel Points: 4438 More actions · Answer 6

These are the situations where having set procedures in place help tremendously. If there is a document outlining the steps to take when something bad happens, the chances of rash action are reduced. A proper change management process, where any change to production has to be tested in acceptance and approved prior to being placed in production, also helps prevent disasters from getting worse.

SQLRNNR SSC Guru Points: 281334 More actions · Answer 7

I agree with the editorial. It is important to remain cool under pressure. That task is much more difficult when you have 30 managers watching over your back. However, it is at that time that remaining cool is much more necessary.

Jason...AKA CirqueDeSQLeil
_______________________________________________
I have given a name to my pain...MCM SQL Server, MVP
SQL RNNR
Posting Performance Based Questions - Gail Shaw[/url]
Learn Extended Events

Elliott Whitlow SSC Guru Points: 102402 More actions · Answer 8

I too agree with the editorial..

It is far too often WAY MORE damaging to jump into a fix without proper consideration than it is to stop and think about it.

Case in point, a recent discussion here about a database that was suspect and some questionable advice on how to fix it. That advice failed to cover WHY the database was suspect and that a little bit of research BEFORE action was REQUIRED.

And when management is squealing about how bad they need this fixed remaining calm and getting it fixed RIGHT THE FIRST TIME helps give them the confidence to let you do your job, not always but it helps.

And I've had a stack of people in my cubie before just watching me type, to the point I turned around and said, "listen, I'm going as fast as I can, you standing there isn't going to make this go any faster. I will let you know when I know something." Which was the subtlest way I could say SCRAM, I'M WORKING HERE..

CEWII

Steve Jones - SSC Editor SSC Guru Points: 736494 More actions · Answer 9

I've actually had to tell a VP to "go away" when I was managing. It was bad enough that I would go in to check on DBAs. Having him walk in was like asking for trouble.

SQLRNNR SSC Guru Points: 281334 More actions · Answer 10

When people start congregating, I tell them I will let them know what the status is with regular email updates. Then I crank up the music 😀

Jason...AKA CirqueDeSQLeil
_______________________________________________
I have given a name to my pain...MCM SQL Server, MVP
SQL RNNR
Posting Performance Based Questions - Gail Shaw[/url]
Learn Extended Events

Elliott Whitlow SSC Guru Points: 102402 More actions · Answer 11

Tunage, great way to drown out the assembled masses..

CEWII

Trey Staker SSCarpal Tunnel Points: 4796 More actions · Answer 12

Excellent Editorial. Been there, done that, have the t-shirt. Both as the DBA or engineer taking the heat triaging the situation and as the management accessing what went wrong and trying to give space to the tech on the problem.

DBA's are usually, not always, people that management trust. We are put in charge of their data for a reason. D/R practice is something that every DBA should be doing often to prepare for situations like this. Most of us are constantly studying most of which is really preparation for when we are called on to react.

When things go wrong panicking is something we must avoid. Management will breath down your neck, they may even get in your face. Often times they're feeling the heat from the situation from the people who are affected by the outage. As you calm yourself remember that this is what you've been preparing for. This is what the hours of study and practice were for. Sometimes 100's of hours of prep boil down to a few minutes of triage followed by seconds of reaction. Look before you leap. Make sure you're taking the appropriate steps and double, triple check what you're doing before you do it.

Another thing that has helped me is to get a co-worker or boss to sheild you from the management distractions while you triage. At one job the exchange admin and I had an agreement before hand to do this when needed.

---------------------------------------------------------------------
Use Full Links:
KB Article from Microsoft on how to ask a question on a Forum

GSquared SSC Guru Points: 260824 More actions · Answer 13

CirquedeSQLeil (3/16/2010)
When people start congregating, I tell them I will let them know what the status is with regular email updates. Then I crank up the music 😀

This works particularly well if your musical tastes include things that others aren't likely to appreciate, or if you're tone-deaf.

For example, if you fire up Yanni, or disco, or anything by "boy band of the hour" (whatever they are), most people will avoid your desk like the plague.

If you don't want to go quite that far, classical music works, and you can claim it helps you concentrate. Opera, if it's in a foreign language, works well for that too. Doesn't have to actually help, just has to be sleep-inducing or "sophisticated".

Anyone who still wants to hang around after you fire up Wagner or Brahms can usually be chased away with a little Country/Western. Follow that with "My Lovely Lady Humps" (whatever the real name of that song is), or anything by Lady Gaga.

Feeding the titles of really bad 70s/80s/90s bands into Pandora can accomplish all of this quite efficiently. Call it your "Crowd Control" playlist.

If self-torture isn't your style, you can always turn it back off once everyone has either left or fallen asleep.

Of course, you take the chance that you won't have any friends left, but at least the space around your desk will be cleared out so you can work.

Disclaimer: I'm not responsible for any brain damage caused by using this tactic. Use it at your own risk. It hasn't been subjected to animal testing, because we couldn't get any volunteers. No humans were harmed in making this film.

- Gus "GSquared", RSVP, OODA, MAP, NMVP, FAQ, SAT, SQL, DNA, RNA, UOI, IOU, AM, PM, AD, BC, BCE, USA, UN, CF, ROFL, LOL, ETC
Property of The Thread

"Nobody knows the age of the human race, but everyone agrees it's old enough to know better." - Anon

Elliott Whitlow SSC Guru Points: 102402 More actions · Answer 14

I just put my headphones on and act like they don't exist..

CEWII