March 31, 2011 at 3:28 pm
TravisDBA (3/31/2011)
Eric M Russell (3/31/2011)
Disaster recovery is actually easier in the modern IT world than it was in times past. One hundred years ago, if the county courthouse burnt to the ground, much of the archived documents would be lost forever with no backup copy.It might be easier as you say, but you would be surprised at how many shops do not plan for it by not even just backing up their databases up on a regular basis. It would astound you. I have seen shops I have gone into in the past that have not backed up their system databases in over a year!!! . When I asked why? the response was "Well we didn't need to, everything works just fine and if it isn't btoke we don't fix it... " :w00t: Absolutely incredible, there are bozos out there in IT like this but there are. . Just because DR has gotten easier doesn't mean people are doing it. 😀
I was not just referring to small mom and pop shops, I was referring to large geographically distributed IT enterprises as well, and it does happens more than you might realize.:-D
"Technology is a weird thing. It brings you great gifts with one hand, and it stabs you in the back with the other. ...:-D"
March 31, 2011 at 3:29 pm
Craig Farrell (3/31/2011)
Tom Bakerman (3/31/2011)
This seems like rehashing old news. I have vague memories from a database class I took in the 80s where we talked about an airlines data center (AA or United or somebody big like that). Like I said, my memory is vague on this, but I believe they talked about Mean Time Between Failure of the disk farm on the order of 5 minutes (translation: there will be a disk failure about every 5 minutes).The technology has changed, but the problems are still there and have to be dealt with.
I'm now picturing some poor kid wandering the halls of the data center with a giant shopping cart of new drives in front of him, just slowly meandering down the aisles looking for red lights with this zombie-fied look on his face at 3 in the morning.
Hehe. But remember, too, that those were the days of the washing machine sized disks. 😛
Another memory I have from that discussion is that the average length of experience of employees was on the order of 10 years.
March 31, 2011 at 3:41 pm
Don't think of it as a failure.
Think of it as a chance to start over without the burden of all that legacy data.
March 31, 2011 at 5:35 pm
Fortunately the Cloud will save us from failure. Because we know how bullet-proof telecom infrastructure is. It never fails...except when it storms, or during peak solar activity, or when a fiber-optic cable is dredged, or the provider goes bankrupt, or during DDoS attacks, or...:-D
James Stover, McDBA
March 31, 2011 at 5:42 pm
James Stover (3/31/2011)
Fortunately the Cloud will save us from failure. Because we know how bullet-proof telecom infrastructure is. It never fails...except when it storms, or during peak solar activity, or when a fiber-optic cable is dredged, or the provider goes bankrupt, or during DDoS attacks, or...:-D
No, the true cloud will be the 'magic mesh network' made up by the billions of cell phones that finally have enough speed to communicate everything everywhere without needing ISPs...
... oh, wait, that was Shadowrun...
Never stop learning, even if it hurts. Ego bruises are practically mandatory as you learn unless you've never risked enough to make a mistake.
For better assistance in answering your questions[/url] | Forum Netiquette
For index/tuning help, follow these directions.[/url] |Tally Tables[/url]
Twitter: @AnyWayDBA
April 1, 2011 at 5:27 am
I think most IT people know they should be planning for failure, but until they have their first one it can hard to really take it seriously and act accordingly. While I obviously didn't think so at the time, looking back I'm glad I had a major failure resulting in substantial data loss fairly early in my career, since that really helped focus the mind! These days I always assume the worst and plan for it.
What you do though at the end of the day is down to assessing the risk and the cost of the potential outcome. For instance, where I work we have some systems that are reasonably critical, they have decent hardware, lots of monitoring, hardware raid, and we maintain a complete set of spare parts just in case. We know the instant something is wrong and can hopefully repair it in a minimal amount of time. We have other systems that just do various non-critical tasks that run on old hardware with no spares, minimal monitoring etc. If they die then it's a pain, but if they're offline for a few days while we build a replacement there's no big issue, and the additional cost of using better kit wouldn't make economic sense.
I agree with Travis though, so many of our clients don't think backups are important (until they lose something of course), and we even had one who literally told us they didn't need backups as if the data was lost they'd just re-type the documentation again! And yes, we did enjoy quoting it back to him when they did lose some data and couldn't recover it.
April 1, 2011 at 7:23 am
Keith Langmead (4/1/2011)
I think most IT people know they should be planning for failure, but until they have their first one it can hard to really take it seriously and act accordingly. While I obviously didn't think so at the time, looking back I'm glad I had a major failure resulting in substantial data loss fairly early in my career, since that really helped focus the mind! These days I always assume the worst and plan for it.What you do though at the end of the day is down to assessing the risk and the cost of the potential outcome. For instance, where I work we have some systems that are reasonably critical, they have decent hardware, lots of monitoring, hardware raid, and we maintain a complete set of spare parts just in case. We know the instant something is wrong and can hopefully repair it in a minimal amount of time. We have other systems that just do various non-critical tasks that run on old hardware with no spares, minimal monitoring etc. If they die then it's a pain, but if they're offline for a few days while we build a replacement there's no big issue, and the additional cost of using better kit wouldn't make economic sense.
I agree with Travis though, so many of our clients don't think backups are important (until they lose something of course), and we even had one who literally told us they didn't need backups as if the data was lost they'd just re-type the documentation again! And yes, we did enjoy quoting it back to him when they did lose some data and couldn't recover it.
Yes, this is all too true. I think there are even some buzzwords for it -- normalcy bias, willful blindness, etc. There definitely seems to be a part of the human brain that does not want to think about the possibility of disaster, and it takes very hard work to overcome it. I know for me it's a daily battle. But I try to learn as much as I can about best practices and tell myself they will pay off in the end.
- webrunner
-------------------
A SQL query walks into a bar and sees two tables. He walks up to them and asks, "Can I join you?"
Ref.: http://tkyte.blogspot.com/2009/02/sql-joke.html
April 1, 2011 at 9:07 am
Craig Farrell (3/31/2011)
Tom Bakerman (3/31/2011)
This seems like rehashing old news. I have vague memories from a database class I took in the 80s where we talked about an airlines data center (AA or United or somebody big like that). Like I said, my memory is vague on this, but I believe they talked about Mean Time Between Failure of the disk farm on the order of 5 minutes (translation: there will be a disk failure about every 5 minutes).The technology has changed, but the problems are still there and have to be dealt with.
I'm now picturing some poor kid wandering the halls of the data center with a giant shopping cart of new drives in front of him, just slowly meandering down the aisles looking for red lights with this zombie-fied look on his face at 3 in the morning.
It's not that bad, but I know a large company, with quite a few SANs, and replacing disk drives is a full time job. It's not 24x7, and they have lots of hot spares to bring online automatically, but someone's job every day is to order drives, unpack/pack up drives, and walk around replacing them.
April 1, 2011 at 9:08 am
Michael Valentine Jones (3/31/2011)
Don't think of it as a failure.Think of it as a chance to start over without the burden of all that legacy data.
ROFL, that is outstanding! 😀
Can I use that quote?
April 1, 2011 at 9:10 am
TravisDBA (3/31/2011)
I was not just referring to small mom and pop shops, I was referring to large geographically distributed IT enterprises as well, and it does happens more than you might realize.:-D
I completely agree with this. All those "department servers" often run without backups. What's even worse is I've seen lots of software places, building commercial software, where they don't backup the dev servers. That's their manufacturing plant, and I've seen some of them lose a decent amount of work because of it.
April 1, 2011 at 9:23 am
Steve Jones - SSC Editor (4/1/2011)
TravisDBA (3/31/2011)
I was not just referring to small mom and pop shops, I was referring to large geographically distributed IT enterprises as well, and it does happens more than you might realize.:-DI completely agree with this. All those "department servers" often run without backups. What's even worse is I've seen lots of software places, building commercial software, where they don't backup the dev servers. That's their manufacturing plant, and I've seen some of them lose a decent amount of work because of it.
Exactly, also good point on the development backups as well.. I saw a case once where development lost a complete block change. Heads rolled on that one. The reason I survived it was I had an email recommending dev backups to the management, but it was considered as not needed because they could not afford to pop for the disk space for dev in their budget at the time. They sure changed their tune on that quick. It only has to happen ONCE (they never really get that), and remember, e-mail is the best CYA in the political dog-eat-dog world of IT. 😀
"Technology is a weird thing. It brings you great gifts with one hand, and it stabs you in the back with the other. ...:-D"
April 1, 2011 at 9:27 am
Hi Steve,
Interesting editorial, this, as it connects failure to probability.
In the end, we all make up the statistics; I just finished 'The Black Swan' (not about the movie!) by Nassim Taleb.
It is worth to check this out, as it shows a refreshing view on how to look at probability as we like to use the concept in models, versus how probability touches us in the real world.
regards,
Hans
April 1, 2011 at 9:32 am
Steve Jones - SSC Editor (4/1/2011)
Michael Valentine Jones (3/31/2011)
Don't think of it as a failure.Think of it as a chance to start over without the burden of all that legacy data.
ROFL, that is outstanding! 😀
Can I use that quote?
Agreed. That was hilarious.
- webrunner
-------------------
A SQL query walks into a bar and sees two tables. He walks up to them and asks, "Can I join you?"
Ref.: http://tkyte.blogspot.com/2009/02/sql-joke.html
April 1, 2011 at 11:12 am
H-122523 (4/1/2011)
Hi Steve,Interesting editorial, this, as it connects failure to probability.
In the end, we all make up the statistics; I just finished 'The Black Swan' (not about the movie!) by Nassim Taleb.
It is worth to check this out, as it shows a refreshing view on how to look at probability as we like to use the concept in models, versus how probability touches us in the real world.
regards,
Hans
Thanks and I'll check it out. My wife wants to see it, so I have an excuse to rent it.
It's not that you worry about the probability of failure, but assessing that over time, or in a large environment during a relatively short time, failure is inevitable.
April 1, 2011 at 11:36 am
Steve Jones - SSC Editor (4/1/2011)
Michael Valentine Jones (3/31/2011)
Don't think of it as a failure.Think of it as a chance to start over without the burden of all that legacy data.
ROFL, that is outstanding! 😀
Can I use that quote?
Yes, be my guest.
Viewing 15 posts - 16 through 30 (of 30 total)
You must be logged in to reply to this topic. Login to reply