February 1, 2017 at 7:09 am
More proof of why you need to test your RESTORE process as much as your backup process:
http://www.theregister.co.uk/2017/02/01/gitlab_data_loss/
4.5GB of 300GB of production data left after a rm -rf command was run in the wrong place...
Poorly documented and fragile scripts for replication, backups that don't appear to have worked, all sorts of problems...
February 1, 2017 at 7:11 am
Brandie Tarvin - Wednesday, February 1, 2017 7:04 AMJust not the one that will cause people to come after you with pitchforks and torches...
Picky, picky, picky.
:hehe:
Well, the torches do tend to set off the sprinkler system, and water and servers don't usually mix well, so...
Of course, if someone brings hot dogs and marshmallows, you can have a cookout with the torches by putting the munchies on the pitchfork tines...
February 1, 2017 at 7:15 am
jasona.work - Wednesday, February 1, 2017 7:01 AMWhile this isn't entirely bad advice, I might lean towards getting the OK to stand up your own "practice" box, whether it be a virtual machine or a physical machine. Something that if you break it, it doesn't stop others from working. Maybe, if given the OK, put copies of your database(s) on it so you've got "real" data to work with.Then, break it, and break it hard.
Practice taking backups and restoring them.
Use an editor to manually edit a database file to corrupt it, then try to fix it.
Break the permissions on the various directories used by SQL Server to learn how to figure out that's what happened (SQL error logs, Windows event logs are your first stops)
But as Brandie said, *BREAK* that box.Just not the one that will cause people to come after you with pitchforks and torches...
^THIS
I have SQL server installed locally on my laptop, I also have a couple VM's in our environments. Learned and tested many things this way and overall it really helps your confidence level.
SIDE NOTE: Being able to just remove certain quoted responses in a reply with just a single click is pretty damn cool.
February 1, 2017 at 7:27 am
Bravery: https://about.gitlab.com/2017/02/01/gitlab-dot-com-database-incident/
They had a live Google Doc open as they worked through the issues.
February 1, 2017 at 7:31 am
Y.B. - Wednesday, February 1, 2017 7:15 AM--SIDE NOTE: Being able to just remove certain quoted responses in a reply with just a single click is pretty damn cool.
Yep, I really like that feature too. Makes it easier to avoid reply-bloat.
The absence of evidence is not evidence of absence.
Martin Rees
You can lead a horse to water, but a pencil must be lead.
Stan Laurel
February 1, 2017 at 7:37 am
Phil Parkin - Wednesday, February 1, 2017 7:31 AMY.B. - Wednesday, February 1, 2017 7:15 AM--SIDE NOTE: Being able to just remove certain quoted responses in a reply with just a single click is pretty damn cool.
Yep, I really like that feature too. Makes it easier to avoid reply-bloat.
Just wanted to add my two bits. I am generally impressed by the new look. Found the remove quote while doing the word of the day. Very nice.
February 1, 2017 at 7:46 am
Steve Jones - SSC Editor - Wednesday, February 1, 2017 7:26 AMBravery: https://about.gitlab.com/2017/02/01/gitlab-dot-com-database-incident/They had a live Google Doc open as they worked through the issues.
Stupidity - "out of 5 backup/replication techniques deployed none are working reliably or set up in the first place"
February 1, 2017 at 8:17 am
BrainDonor - Wednesday, February 1, 2017 7:46 AMSteve Jones - SSC Editor - Wednesday, February 1, 2017 7:26 AMBravery: https://about.gitlab.com/2017/02/01/gitlab-dot-com-database-incident/They had a live Google Doc open as they worked through the issues.
Stupidity - "out of 5 backup/replication techniques deployed none are working reliably or set up in the first place"
That would be the "winner" for the day. :sick: I hope the lesson is learned by many - test the backups.
February 1, 2017 at 8:43 am
Brandie Tarvin - Wednesday, February 1, 2017 4:36 AMDon't take this the wrong way, but that bit I bolded is NOT a good thing. You need to break the Dev box. Seriously. Hammer that sucker until you have to rebuild either the instance or the databases. Bring on a major disaster because you need the experience on fixing the major disaster.I'm not kidding here. This is the most valuable experience a DBA can have, especially when they are starting out. Once you get it under your belt, you can move forward without worries because you KNOW without doubt you can fix the major disasters.
EDIT: Of course, you might want to warn your boss and coworkers that's what you're doing before you do it. Just so they don't panic when the Dev box goes down. @=)
Small disagreement there.
Don't break dev. The dev server is essentially a production environment to the dev team.
Build a server, configure it, break it and then fix it.
Gail Shaw
Microsoft Certified Master: SQL Server, MVP, M.Sc (Comp Sci)
SQL In The Wild: Discussions on DB performance with occasional diversions into recoverability
February 1, 2017 at 8:55 am
GilaMonster - Wednesday, February 1, 2017 8:43 AMBrandie Tarvin - Wednesday, February 1, 2017 4:36 AMDon't take this the wrong way, but that bit I bolded is NOT a good thing. You need to break the Dev box. Seriously. Hammer that sucker until you have to rebuild either the instance or the databases. Bring on a major disaster because you need the experience on fixing the major disaster.I'm not kidding here. This is the most valuable experience a DBA can have, especially when they are starting out. Once you get it under your belt, you can move forward without worries because you KNOW without doubt you can fix the major disasters.
EDIT: Of course, you might want to warn your boss and coworkers that's what you're doing before you do it. Just so they don't panic when the Dev box goes down. @=)
Small disagreement there.
Don't break dev. The dev server is essentially a production environment to the dev team.Build a server, configure it, break it and then fix it.
The dev box is actually our own BI dev system so we've actually got a pretty free rein in there. There's only my boss and me that use it.
I think flattening it, building it from the metal up, breaking it and fixing it will all be on the agenda for future exercises. We've also put a request in for a couple of VM's for practice purposes.
John (and anybody else in the Leeds area come to that), a couple of pints after work sounds like a splendid idea. I'm actually staying in Leeds for the week. I'm only down the road really but I couldn't be bothered with, potentially, a four hour round trip in the glorious M62 traffic.
How to post a question to get the most help http://www.sqlservercentral.com/articles/Best+Practices/61537
February 1, 2017 at 9:20 am
jasona.work - Wednesday, February 1, 2017 7:09 AMMore proof of why you need to test your RESTORE process as much as your backup process:
http://www.theregister.co.uk/2017/02/01/gitlab_data_loss/
4.5GB of 300GB of production data left after a rm -rf command was run in the wrong place...
Poorly documented and fragile scripts for replication, backups that don't appear to have worked, all sorts of problems...
More proof that replication is NOT a disaster recovery solution.
February 1, 2017 at 9:24 am
BWFC - Wednesday, February 1, 2017 8:55 AMGilaMonster - Wednesday, February 1, 2017 8:43 AMBrandie Tarvin - Wednesday, February 1, 2017 4:36 AMDon't take this the wrong way, but that bit I bolded is NOT a good thing. You need to break the Dev box. Seriously. Hammer that sucker until you have to rebuild either the instance or the databases. Bring on a major disaster because you need the experience on fixing the major disaster.I'm not kidding here. This is the most valuable experience a DBA can have, especially when they are starting out. Once you get it under your belt, you can move forward without worries because you KNOW without doubt you can fix the major disasters.
EDIT: Of course, you might want to warn your boss and coworkers that's what you're doing before you do it. Just so they don't panic when the Dev box goes down. @=)
Small disagreement there.
Don't break dev. The dev server is essentially a production environment to the dev team.Build a server, configure it, break it and then fix it.
The dev box is actually our own BI dev system so we've actually got a pretty free rein in there. There's only my boss and me that use it.
I think flattening it, building it from the metal up, breaking it and fixing it will all be on the agenda for future exercises. We've also put a request in for a couple of VM's for practice purposes.
That sounds like an excellent plan. Also, this is a good time to start making notes about what your recovery strategy should be (versus what it actually is) and to get other people involved in impromptu DR exercises that might keep your company from pulling a GitLab. I mean, testing the tools is important. Right?
February 1, 2017 at 1:23 pm
Steve Jones - SSC Editor - Wednesday, February 1, 2017 7:26 AMBravery: https://about.gitlab.com/2017/02/01/gitlab-dot-com-database-incident/They had a live Google Doc open as they worked through the issues.
Too often have I been called in after such a blunder, unplug the storage, run DD recover, storage level recovery etc. nothing mitigates such mishaps but one can often "half the damage".
😎
Brandies point was a good one, five to one DR failure is worth looking at!
February 1, 2017 at 3:03 pm
Happy, delighted, humbled and overall gleeful to announce that I'm awarded the Microsoft Data Platform MVP award.
Yes, it's not January, April, July or October. Microsoft has changed the cycle for new MVP awardees to a monthly basis.
Renewal is now once a year at July.
Need an answer? No, you need a question
My blog at https://sqlkover.com.
MCSE Business Intelligence - Microsoft Data Platform MVP
February 1, 2017 at 3:28 pm
Koen Verbeeck - Wednesday, February 1, 2017 3:03 PMHappy, delighted, humbled and overall gleeful to announce that I'm awarded the Microsoft Data Platform MVP award.
Yes, it's not January, April, July or October. Microsoft has changed the cycle for new MVP awardees to a monthly basis.
Renewal is now once a year at July.
Congrats and well deserved!!!
_______________________________________________________________
Need help? Help us help you.
Read the article at http://www.sqlservercentral.com/articles/Best+Practices/61537/ for best practices on asking questions.
Need to split a string? Try Jeff Modens splitter http://www.sqlservercentral.com/articles/Tally+Table/72993/.
Cross Tabs and Pivots, Part 1 – Converting Rows to Columns - http://www.sqlservercentral.com/articles/T-SQL/63681/
Cross Tabs and Pivots, Part 2 - Dynamic Cross Tabs - http://www.sqlservercentral.com/articles/Crosstab/65048/
Understanding and Using APPLY (Part 1) - http://www.sqlservercentral.com/articles/APPLY/69953/
Understanding and Using APPLY (Part 2) - http://www.sqlservercentral.com/articles/APPLY/69954/
Viewing 15 posts - 57,301 through 57,315 (of 66,738 total)
You must be logged in to reply to this topic. Login to reply