Post reply

complex problem

  • February 3, 2012 at 10:24 am

    #267647

    I have a 2 node active passive cluster. They are dell 810 with 4 6core procs, 128 gigs ram and 10TBs of San storage. A week ago my server crashed and I manually had to move to node 2. Before the crash I had an 11-30 ms response time. 11 being the norm and 30 being spikes. After the crash my response times start low and slowly build over 4-5 hours int o the range of 600 - 1800 ms, and server comes to a crawl. This is the same on node 1 and node 2. I have been working with dell on the problem and we have no answers at this point.

    Has anybody seen anything like this.

    any direction would be greatly appreciated.

    any information you want I will gladly provide to get an idea where to fix the problem.

    [font="Tahoma"]John Burris:hehe:
    MCITP Database Administrator[/font]

  • February 5, 2012 at 2:14 pm

    #1442852

    I have no idea and I think you're doing the right thing by working with DELL on this. Hopefully my response will also serve to "bump" your post back up where someone might see it and answer.

    --Jeff Moden

    RBAR is pronounced "ree-bar" and is a "Modenism" for Row-By-Agonizing-Row.
    First step towards the paradigm shift of writing Set Based code:
    ________Stop thinking about what you want to do to a ROW... think, instead, of what you want to do to a COLUMN.

    Change is inevitable... Change for the better is not.

    Helpful Links:
    How to post code problems
    How to Post Performance Problems
    Create a Tally Function (fnTally)

  • February 5, 2012 at 5:59 pm

    #1442873

    What wait types are you seeing on the slow queries?

    Gail Shaw
    Microsoft Certified Master: SQL Server, MVP, M.Sc (Comp Sci)
    SQL In The Wild: Discussions on DB performance with occasional diversions into recoverability

    We walk in the dark places no others will enter
    We stand on the bridge and no one may pass

  • February 6, 2012 at 8:33 am

    #1443213

    John Burris (2/3/2012)

    I have a 2 node active passive cluster. They are dell 810 with 4 6core procs, 128 gigs ram and 10TBs of San storage. A week ago my server crashed and I manually had to move to node 2. Before the crash I had an 11-30 ms response time. 11 being the norm and 30 being spikes. After the crash my response times start low and slowly build over 4-5 hours int o the range of 600 - 1800 ms, and server comes to a crawl. This is the same on node 1 and node 2. I have been working with dell on the problem and we have no answers at this point.

    Has anybody seen anything like this.

    any direction would be greatly appreciated.

    any information you want I will gladly provide to get an idea where to fix the problem.

    Get a performance tuning consultant on board to look for things. Server settings, database settings, MANY hardware/firmware/driver potential causes, etc.

    Best,
    Kevin G. Boles
    SQL Server Consultant
    SQL MVP 2007-2012
    TheSQLGuru on googles mail service

  • February 6, 2012 at 11:10 am

    #1443346

    Is this a Windows 2008 and SQL2008 (r2) server?

    Do you know if the windows patch levels were the same between nodes?

    Were the security policy changes mirrored such as enabling Lock Pages in Memory?

  • February 7, 2012 at 7:58 am

    #1443796

    Case Closed.

    It turned out that when the Fan crashed, the tech asked me to apply an update for the system that was supposed to monitor the fan better. This update caused the system as a whole to slowdown. When we got the correct patch and windows version in sync, the server came back to normal speeds.

    Thanks for taking the time to look at this.

    [font="Tahoma"]John Burris:hehe:
    MCITP Database Administrator[/font]

  • February 7, 2012 at 9:18 am

    #1443898

    Too funny! 🙂 People used to laugh at me when they asked me what the first thing I'd do as a DBA in the morning (besides grabbing coffee). My answer was, "Make a trip through the server room to see if all the right lights were blinking, touch each box to see how warm it was, listen for a change in fan noise, and sniff for anything that smelled hot." 😛

    --Jeff Moden

    RBAR is pronounced "ree-bar" and is a "Modenism" for Row-By-Agonizing-Row.
    First step towards the paradigm shift of writing Set Based code:
    ________Stop thinking about what you want to do to a ROW... think, instead, of what you want to do to a COLUMN.

    Change is inevitable... Change for the better is not.

    Helpful Links:
    How to post code problems
    How to Post Performance Problems
    Create a Tally Function (fnTally)

Viewing 7 posts - 1 through 6 (of 6 total)

You must be logged in to reply to this topic. Login to reply