Troubleshooting

  • aureolin (4/2/2009)


    In an emergency you must remember to panic!!

    Well, not really, but people who *are* panicking often get really angry at your failure to panic along with them. They see your calmness not as competence in the face of an emergency, but as a "failure" on your part to "understand the seriousness" of the situation. If you're not careful (and sometimes even if you are), you can suddenly find yourself defending your actions and your attitude from a frustrated and angry person instead of solving the problem.

    In my experience, the best bet is to remove the panicky people from the area where you are trying to problem-solve. This can help break the escalation of their panic (and give them a chance to calm down) and remove a fairly serious distraction while you're in an emergency situation.

    I disagree. In almost every case I've been in, as long as people saw that I understood the urgency, they became calmer. I tend to do something to the effect of:

    - Say, "All right, let's stay calm or become calm because emotion is going to cloud our thinking and we need to solve this as fast as possible. That means we've got to be able to think as clearly as possible."

    - Then say, "Okay, very quickly, what do we know?"

    - Then say, "Okay, shortly and succinctly, what have tried and what were the results?"

    I use words like "very quickly" and "shortly and succinctly" to communicate that I understand the urgency and gravity of the situation but that we're in control of the situation. And if I know that they are the types who need/demand status reports, I then set a schedule. Usually on the hour, of if necessary, every 15 minutes. I make sure that I tell them if we find something significant, they'll get a status report sooner. That seems to stop the pacing outside the cube.

    K. Brian Kelley
    @kbriankelley

  • Hi Mike

    No need to appologize did nothing wrong, I just find that sometimes people look at SQL Server as the little brother and I get very frustrated because of this(not saying you did or the other guy ). It's alsways DB2 this and Oracle that and sometimes I get a bit emotional around those comments and statements.... Microsoft has made excessive footprints in the OLTP\OLAP market ....and according to the latest TCP benchmarks ...SQL is a fantastic product and in some benchmarks even outclass the other one's but DB2 and Oracle are fantastic products as well......so I will calm myself down now :blush:

    The 10 min an 1 hour comment:

    I am refering to save time when there is an outage. Good methodology and skillsets can always save time on fixing the problem. And you could use a methodology to also indicate skill shortages(although this is up to the individual sometimes if he\she wants to advance in their career).

    One of the gents also made a comment about documentation when problems are experienced...so true ...we all feel the pain and sometimes we are also guilty of it ourselfves..

    It is also dicipline that needs to be applied for documentation and there is no excuse for this .......

    Once again well written article.

  • and off course there is a difference in Red Cross junior camp first aid, Paramedic first aid and combat first aid πŸ˜€

    According to their trained skills, you may have a better chance of:

    - surviving their actions

    - surviving the incident

    - quality of life afterward.

    In some cases, it is better to call for a paramedic.

    Johan

    Learn to play, play to learn !

    Dont drive faster than your guardian angel can fly ...
    but keeping both feet on the ground wont get you anywhere :w00t:

    - How to post Performance Problems
    - How to post data/code to get the best help[/url]

    - How to prevent a sore throat after hours of presenting ppt

    press F1 for solution, press shift+F1 for urgent solution πŸ˜€

    Need a bit of Powershell? How about this

    Who am I ? Sometimes this is me but most of the time this is me

  • mike_walsh (4/2/2009) I agree doesn't translate to a fire scene but so far it has worked for me in SQL and the Ambulance.

    I thought about this at lunch and I actually think that while the analogy may not fit a lot of the steps still fit into a "rely on training" type of situation. You may not have time to go to checklists like I blogged about here comparing doctors,chefs,pilots and DBAs. You may also end up skipping some steps but the same general pattern works in those look and act type scenarios. I also hope that whoever has incident command is thinking through them, the IC at a fire is supposed to work with the other members of command (Safety, Operations, Medical, etc) and make informed decisions. Even the operations leads in the fire should practice some of the below. For a sturcture fire:

    1. Gather initial information - What type of structure? What other hazards are around? Live high voltage lines lying in a puddle? People inside or just a structure? Propane tank? Any law enforcement concerns? Traffic safety hazards? Water supply?

    2. Prepare your mind - En route to the call, what is your job? You are going back to your training but what role are you in, think through it, know it and prepare to do it.

    3. Work as a team - So very important in a fire. You don't have to think about this, training has hopefully beaten it into you but you go in as a team and, Lord willing, come out as a team.

    4. Plan your attack - Victims trapped? You will be more aggressive (but still not stupid). Just property? Is the fire contained enough to send a crew in? Where will you approach from? Where will you put emergency ladders for dumping out of the house in a pinch? How is the structural integrity?

    5. Don’t be afraid of asking for help - Confirm a command if you aren't sure, bring in mutual aid early, etc.

    6. Formulate a problem statement and verify it with all parties! - This one is sort of easy.. There's some red stuff that needs wet stuff πŸ˜‰ This is sort of an assumed portion here and probably superflous.

    a. We also looked at the entire picture and didn’t develop tunnel vision. - Were you so focused on the flames or screams that you missed you were about to kill yourself with that pole to house electrical line flapping in the breeze?

    7. Remain Calm - Big Plus on the fire scene. Adrenaline is flowing in huge buckets already. Having a chief panic (and I have seen it) and get angry/flustered does no good. The crews will do stupid things and someone risks getting hurt (or worse). Stay in control.

    8. Understand priority and if the issue is stable or declining - Speaks for itself in a fire, right? At what point do you go from offensive to defensive attack? Do you even skip offensive attack (in spite of reported entrapment) because it is far too dangerous?

    9. Anticipate changes and plan ahead for worsening – RIT teams (Rapid Intervention Teams) are becoming more and more standard at fire scenes. It's a team all geared up, ready to go in and rescue downed firefighter(s). That is what they train for and they are outside waiting for that purpose. You have ambulances there even if no victims in case a firefighter gets hurt. You continue to monitor the house and surroundings looking for new hazards.

    10. Use all of the information – You are using everything from the above steps in making your plans.

    11. Documentation – Even here, insurance companies, training "opportunities", etc.

    12. Clean up and preparation for next call – You hope there a lot of probationary firefighters around to roll hose. The last thing a bunch of exhausted firefighters want to do is clean up after themselves πŸ˜‰

    I agree, a lot of these steps are look and act type steps but even here you are not just looking only. Yes you are relying on training a LOT more but even still if you train for firecalls with a consistent methodology (which is what Incident Management Systems is for command/control aspects) it really does become just "another problem" to troubleshoot. Decisions come a lot faster and less alternative options are gone through or gone through a lot faster but I think the methodology does work.

    __________________________________________________

    Mike Walsh
    SQL Server DBA
    Blog - www.straightpathsql.com/blog |Twitter

  • ALZDBA (4/2/2009)


    and off course there is a difference in Red Cross junior camp first aid, Paramedic first aid and combat first aid πŸ˜€

    According to their trained skills, you may have a better chance of:

    - surviving their actions

    - surviving the incident

    - quality of life afterward.

    In some cases, it is better to call for a paramedic.

    You got that right πŸ™‚ Know when to ask for help and know your own limitations. No matter the situation. You could be on a fire scene and a captain or chief says "Get that ladder up there!" and you've never operated that departments ladder truck... Much better to ask for help than be embarassed and try and figure it out on the fly.

    You could be trying to figure out a hardware problem and see that something in the SAN "feels wrong" but you aren't trained there, only know casual information about the SAN, etc. Rather than just try and change it blindly, escalate... Get help.

    __________________________________________________

    Mike Walsh
    SQL Server DBA
    Blog - www.straightpathsql.com/blog |Twitter

  • haha. I wonder how many people got the "Johnny and Roy" reference. πŸ™‚ I loved Emergency.

  • DavidMcAfee (4/2/2009)


    haha. I wonder how many people got the "Johnny and Roy" reference. πŸ™‚ I loved Emergency.

    πŸ˜€ First to catch it and comment on it anyway. Desoto and Gage. Never really watched it a whole lot except some reruns (showing my young age πŸ˜‰ ) but they are still referred to at least once per training class you seem to take (not sure if that's good or bad).

    __________________________________________________

    Mike Walsh
    SQL Server DBA
    Blog - www.straightpathsql.com/blog |Twitter

  • This is a clever article and discussion and fun to participate and read. The analogy and abstraction hits some marks and stimulates thinking. I'm want to say that nothing is final and absolute and that people change, laws change, priorities change but we still need rules and laws and sometimes they fit and sometimes they don't. But some things are universal and never change. And some of those are in this article.

  • Mike, this was a good article, quite well written and thought out. It's good to think of things differently in this manner, which helps to deliver the sense of urgency as well as empathy for users (patients/family members).

  • Hello

    Im from peruvian, and i found interesting the article, but there are some terms that is difficult to understand, like "ShotGun", more or less what it is means.

    Good Post.

    Thank You

  • esteban alvino quispe (4/3/2009)


    Hello

    Im from peruvian, and i found interesting the article, but there are some terms that is difficult to understand, like "ShotGun", more or less what it is means.

    Good Post.

    Thank You

    Thanks Esteban,

    I apologize for not being more international friendly in my post. To simplify: a Shotgun is the kind of gun (firearm) that shoots (fires) a series of small pellets or large slugs. It's used in bird hunting with the smaller pellets and multiple pellets loaded at one time. This way when it is fired, there is a better chance of hitting the target. I used the term to say throwing a lot of things at the problem and hoping one of them hits it πŸ™‚

    http://en.wikipedia.org/wiki/Shotgun

    __________________________________________________

    Mike Walsh
    SQL Server DBA
    Blog - www.straightpathsql.com/blog |Twitter

  • mike_walsh (4/3/2009)


    esteban alvino quispe (4/3/2009)


    Hello

    Im from peruvian, and i found interesting the article, but there are some terms that is difficult to understand, like "ShotGun", more or less what it is means.

    Good Post.

    Thank You

    Thanks Esteban,

    I apologize for not being more international friendly in my post. To simplify: a Shotgun is the kind of gun (firearm) that shoots (fires) a series of small pellets or large slugs. It's used in bird hunting with the smaller pellets and multiple pellets loaded at one time. This way when it is fired, there is a better chance of hitting the target. I used the term to say throwing a lot of things at the problem and hoping one of them hits it πŸ™‚

    http://en.wikipedia.org/wiki/Shotgun

    Or this link:

    http://es.wikipedia.org/wiki/Escopeta

    __________________________________________________

    Mike Walsh
    SQL Server DBA
    Blog - www.straightpathsql.com/blog |Twitter

  • Thank you for the article! I have always been quietly grateful that in my IT line of work I didn't have to be responsible for human lives (not directly anyway). The analogy used here cleverly illustrates the parallels and how we can approach our IT jobs with the same calm and structure as do people in much more critical situations. I think the benefits of planning and keeping cool head are the most important ones.

  • Great Article!

    My first career was an EMT working on an ambulance (back in 1980), and never thought about how it applies to what I do now... gave me an interesting perspective...

    Mark

  • That was back in the wild west days of EMS still, wasn't it? πŸ™‚

    I like trying to find lessons that are transferable from career to career or life lesson to career, vice-versa. I'm lazy πŸ˜‰

    __________________________________________________

    Mike Walsh
    SQL Server DBA
    Blog - www.straightpathsql.com/blog |Twitter

Viewing 15 posts - 31 through 44 (of 44 total)

You must be logged in to reply to this topic. Login to reply