October 14, 2010 at 8:58 pm
Comments posted to this topic are about the item Risk Analysis
October 14, 2010 at 10:16 pm
Interesting timing... Our Enterprise Systems team, in conjunction with one of our (DB2) dba's is testing our JDE DR environment this weekend. Happens twice a year - flush through all the logs, do rowcount comparison and run a handful of key reports.
We don't cost the actual spend on DR individually with any clarity. We make out a business case based on the recovery requirements for the cost of SETTING UP. The cost of TESTING the arrangement is part of BAU of the department running the system, and no separate cost provision is made for the test.
At one stage we had a well formalised business continuity planning arrangement in place, where each application was assessed individually based on the cost of an outage (a) internally and (b) if external, then externally and (c) on our reputation. Not too sure how well this is kept up to date, as I am no longer close to the process.
October 14, 2010 at 11:15 pm
I am on the BI end. Disaster recovery is not my problem: I deal with derivered (2nd gen) data only. If I had perfect copies of the originalo 1st-gen source data but my primary sources got wiped out, I still would have to synch up to them.
October 15, 2010 at 1:55 am
We are a large site, but spend nothing (at the moment).
Our managment do talk about it a lot though...
October 15, 2010 at 2:07 am
With disk space getting cheaper and cheaper, I'm considering doing some basic disaster recovery for home.
Copying all music/photos etc onto a big disk and taking it to a relative's house.
That way if my house burns down, I'll still have the photos etc.
October 15, 2010 at 6:18 am
We actually have off-site warehouse space where we have cublicles and phones for our customer service team. A backup AS400 and multiple Windows servers are offsite as well. We backup all of our SQL Server databases, and those backups are backed up themselves with CommVault. In the event of a disaster my job would be to rebuild all of the SQL Server databases so our customers can access their reporting data. Our goal is to be 100% back up and running within 5 days.
October 15, 2010 at 6:51 am
Ben Moorhouse (10/15/2010)
With disk space getting cheaper and cheaper, I'm considering doing some basic disaster recovery for home.Copying all music/photos etc onto a big disk and taking it to a relative's house.
That way if my house burns down, I'll still have the photos etc.
That's interesting. I wonder if I ought to try that with a friend. Copy/encrypt stuff onto DVDs and leave it there. Certainly cheaper than something like Carbonite.
October 15, 2010 at 6:52 am
Saving the data is the easy part.
The biggest part of our DR is questions of how we get the system going: the network, the production control and the customer service features back up and running. That inclues (we have a third party contracted location) detailed infomation on switch configuration, internet DNS changes, phone re-routing, employee contact, contracts for rented equipment, tech support etc, as well as the sequence, and deciding which systems will be brought up immediately and which can be either deferred or (depending on the nature of the disaster) wait until we are back in our facility.
The more you look at it the more intimidating it becomes.
...
-- FORTRAN manual for Xerox Computers --
October 15, 2010 at 6:54 am
Steve, I was actually thinking along the lines of hotswap hard disks!
It's around £50 for a 1TB drive over here, a couple of those would allow you to rotate them regularly to keep your backups current.
October 15, 2010 at 7:02 am
I sure wish we spent more effort, especially after yesterday! I've on the development side, so it was "NMP" (not my problem) but our two-server SQL cluster, connected to an external storage where the data resides, went tango-uniform yesterday. Long story short (from the sys admins) is that the RAID controller on the external storage died and the clusters fought each other for control of the data to the point where the master database got corrupted. Again, I'm not the expert so my explanation might sound a bit hokey. Anyway, they ended up moving everything to a temporary server while the cluster is re-built. Users were down for at least 24 hours. Not fun...
October 15, 2010 at 7:23 am
DR always has to be a priority for business IMO. At my last job, there was discussion as to whether we really "needed" a generator. After all, Power outages aren't THAT long, and even if we went down, a UPS would hold the load long enough for a clean shutdown. Convincing the higher-ups that the money spent on a generator was a good investment was tough since no apparent ROI was there.
A year later there was a massive wind storm in our state. Large chunks of our metropolitan area were without power for up to a week as they scrambled to fix all of the damage, and people were coming into work to shower and clean up since they had no power at home. The location where we were specifically had no power for 3 full days, all business days. If there wasn't a generator in place, we would have had to go completely to our remote DR plan which would have been extremely costly and painful. With a generator, all we had to do was keep it fueled and turn off all non-essential equipment at closing time. It completely paid for itself that day.
DR is never seen as good until you NEED it. If you choose to go without it you are gambling with dice with your business.
October 15, 2010 at 7:37 am
This is regarding how a company should go about quantifying the extent of investment required for disaster recovery. From my experience, the frequency of a company wide network outage tends to be greater than database server outages, although network outages tend to be shorter in duration. So, the next time the network goes down, ask all employees to summarize the extent to which their job or the company's revenue was affected.
For example, when the ecommerce portal went down for 6 hours on Tuesday morning, how did that impact the total number of new orders for the week; did the company actually lose sales, or did the sales volume simply pick up Tuesday evening or Wednesday? If a daily report had to be generated the following day, was it just the minor inconvenience of having to kick it off manually, or was there a measurable loss of revenue? Can Tuesday's report be acurately generated on Wednesday?
In an economy where key personnel are sacrificed in order to make ends meet, a company that actually has a disaster recovery plan in place may choose not to spend that additional $100,000 on efforts to avoid what amounts to temporary inconveniences.
"Do not seek to follow in the footsteps of the wise. Instead, seek what they sought." - Matsuo Basho
October 15, 2010 at 7:40 am
DR and BCP has been a huge focus for us this year. We started this earlier in the year and our project is still not 100% complete. I think from start to finish it will have taken 12+ months to implement. In the end we will have DR for almost all applications and services.
We will have redundancy for all critical applications and data: Email, Database, Management System Applications, Files, Phone system, Web applications and Web sites. In the event of total destruction to our office we will be able to conduct business as normal with support for remote users. Our DR will enable all employees to be geographically dispersed anywhere (in hotels in the event of an evacuation) and still take phone calls and conduct business as "normal".
We had to purchase additional servers and another phone chassis, MPLS and PRI circuits as well as rack-space at a datacenter. We will be utilizing the offsite datacenter as our primary data site with our headquarter location as our failover site. We'll have servers at both facilities and a secondary phone system with additional PRIs and an MPLS Circuit installed so call traffic can be rerouted to the colo site.
We also purchased recovery services that will provide us Internet, Power, Office Space and Hardware in the event of a disaster. All of that equipment is procured already so we are guaranteed it will be delivered.
So far it's been a one-time cost of $35,000 and will be an on-going yearly cost of $31,000. I expect that one time cost to increase by about another $10,000 by the end of next year after we purchase additional laptops.
Email Redundancy (thru hosted services): $3,200/yr
Database Redundancy: $10,300 (additional server and new SQL License)
Web Redundancy: $4,000 (additional server/OS License)
Firewall for additional site: $2,300
Colo Hosting: $14,000 /yr
Data Pipe between facilities: $11,000/yr
Secondary Phone System: $18,500
Recovery Services: $3,000/yr
For us that cost is well worth it. The money we'd lose once our integrity is lost b/c we couldn't service our customers would be far beyond the DR costs. Plus it's huge benefit to our employees. Yes we will stay in business even if all is lost. You will have a job and you can continue to get a paycheck in the middle of a hurricane, or fire or any other crippling disaster. Those are words you want to hear especially in today’s economy.
October 15, 2010 at 7:58 am
We have a great DR plan but there are no equipment costs associated with it, only time. I work in county government, but we are completely decentralized (not my idea and certainly there would be cost savings associated with centralization). My DR server is an old server no longer covered under warrantly, but it is a big workhorse and houses the critical applications along with the DC. Not an optimal setup, but certainly would work in the event of a disaster. It is located in another county department's server room (they have equipment in my server room as well). This trade-off works well and there are no expenses associated with it. We are well prepared in the event that this office becomes a hole in the ground, I am glad to say, though I certainly hope that never happens.
October 15, 2010 at 8:30 am
Annette Ehrlich (10/15/2010)
...We are well prepared in the event that this office becomes a hole in the ground, I am glad to say, though I certainly hope that never happens.
Regarding DR plans, it's also important to remember documentation. In the event that an office should become a "hole in ground", it's not likely those employees working onsite that day will be available to help with the transition... :unsure:
"Do not seek to follow in the footsteps of the wise. Instead, seek what they sought." - Matsuo Basho
Viewing 15 posts - 1 through 15 (of 29 total)
You must be logged in to reply to this topic. Login to reply