Post reply

The DBA Whoops

  • June 16, 2005 at 9:28 am

    #566156

    What about it?

  • June 16, 2005 at 9:43 am

    #566164

    I read this somewhere:

    "Measure twice, cut once."

    Learn it well; it will save your dunkey.


    {Francisco}

  • June 16, 2005 at 9:53 am

    #566169

    Every time this has happened to me, I always blame the software vendors for not correctly implamenting time travel API's. Lets go developers, make my job easier!

  • June 16, 2005 at 9:53 am

    #566168

    So true .

  • June 16, 2005 at 9:54 am

    #566170

    Sorry, I'm not gonna be the first one to implement this.

    Could we fast-forward to tomorrow afternoon ?

  • June 16, 2005 at 10:14 am

    #566195

    Using transactions looks like the best way to prevent such mistakes.

    An alternative would be to use 'set rowcount'. The rowcount value can be 1 more than the number of rows affected. So, if 1 row is to be updated, it can be set to 2. If the return message says 2 rows are affected, then you know you messed up but atleast the damage is limited.

    set rowcount n -- n is the number of rows affected by DML + 1

    update....

    set rowcount 0

     

  • June 16, 2005 at 10:14 am

    #566194

    Here is another story from one of my old job:

    We had a contractor coming in to work .... somehow the contractor did

    delete table

    So he told the manager what happen.

    The manager try to manually type in all 900 + rows one at a time.

    By the time I got to work, the manager try to insert data manaually because he had a print out copy of it.  Let's just say he got about 50 rows in the table.

    I had to convince him that I can get it done faster then him (manager) entering it manually before he agreed that I can restore the database to a new name, and bring back the old data for just that table.

     

    mom

     

  • June 16, 2005 at 10:24 am

    #566202

    I still preffer no damage at all if possible.

    I think I would just have restore without its approval (under tableName_restored), then showed him he was working for nothing .

  • June 16, 2005 at 1:03 pm

    #566239

    Geez!!! All this talk about whoops and guess what happens???

    I was moving two new stored procedures from dev to prod.  I usually use query analyzer to do this, but I was already in EP and figured I'd use a wizard to move them.  Wizard moved them fine.  When I went into the website to test the new functionality, I was getting errors everywhere.  Back to my production database; there is no data in any table!!!

    After running around the building a few times screaming at the top of my lungs, I performed a restore using the backup from last night.  It said everything was restored successfully.  There is still no data in any of the tables!!!   After managing to keep breakfast from decorating my office, I tried a different track.

    I restored to a new DB and all the data shows up; whew. 

    Now I need to go to my test environment and figure out how I got into this mess-up in the first place!!!

    The article was great, but I hope I never have one of those moments again! 

  • June 16, 2005 at 1:07 pm

    #566240

    I can so picture myself doing that .

    Maybe we should redo that disaster contest. Looks like we'd have a few new inputs .

  • June 16, 2005 at 2:23 pm

    #566257

    Good article, and number of valuable lessons come out of this:

    1. Never do an update directly to a production db without first testing this in development or QA environment first and then after validating the result, only run that pasted query in production. Then you won't have to highlight anything.

    2. Always start production action queries in the QA with a BEGIN TRAN. In this way, you can look at the result set before commiting it or simply roll it back. This is a saftety net that should always be used.

    3. Don't highlight production action query code in the QA under any reason. This is what the Clear Window button is for. Only paste in the query you plan on using and only after you have tested and saved it. Never highlight anything in the QA on production.

    4. Never do things in a hurry, or when you are tired. Sometmies, you don't have a choice, but alot of times you do. Also, never rush something into production without testing based on someone telling you to do this no matter what their position. After all. it will not be them that loses their job over this, it will most likely be you. I have seen this too many times in the past.

    Travis Lee Alltop

    SQL Server DBA

  • June 16, 2005 at 6:27 pm

    #566287

    Fantastic article. The great thing about being human we make mistakes and then we learn from them. 

    Be nice if there was an undo command or button in SQL server 🙂

     

  • June 16, 2005 at 7:12 pm

    #566288

    Great article! I've been a DBA for nearly 15 years, and a system administrator for 5 years before that...I've been through this kind of thing several times. The lesson I appreciated most was to "inform someone". I've learned that managers would rather know that something's wrong so they can do damage control rather than have some VP or CIO catch them by surprise.

  • June 17, 2005 at 2:46 am

    #566337

    On Saturday morning last week I picked the phone and this is what I've heard:

    Hello! We've got a serious problem here. I've just ereased all the drives on SQL Server machine.

    This is not a joke. This really happened.

  • June 17, 2005 at 2:51 am

    #566338

    How the heck did he do that without noticing he was screwing something up??

Viewing 15 posts - 31 through 45 (of 62 total)

You must be logged in to reply to this topic. Login to reply