Early last week, my church suffered a lightning strike that did quite a bit of damage (relatively speaking) to computer and media equipment. I spent a lot of time last week during off hours working the issue along with some other very knowledgable folks. We're not out of the woods yet, but we were up and running and were able to cover a funeral last Friday and worship services on Sunday. In thinking about it all, it brought me back to business/disaster recovery. Since a church is a relatively simple model, I figured I would blog about it to hopefully stir some thoughts for more complex situations and conditions.

Understand What You Need and When You Need it By, Before the Disaster

This one should be self-explanatory, but I've seen cases where it has not been done. I'm the junior high youth pastor at my church and we're not very heavily computerized like some churches. So no one had really considered this thoroughly for us. We carry cell phones if members need to reach us. As long as we have a building with seating, we can hold most of the services we "provide." So our essential recovery is rather limited in scope. But if you're a business of any size, you need to do this before a disaster does hit. In our case we made a quick evaluation of what was critical for services. Getting all of our lighting back was essential. Sound and A/V for the funeral and services was most pressing after that. Establishing network connectivity, especially Internet connectivity for a couple of the systems followed. Everything else was after those key priorities.

Assess the Damage

After the lightning strike, one of the first things we needed to do was assess the damage. Actually, we took a slightly different approach, in that we assessed what "systems" didn't work. Knowing what is damaged tends to be a little trickier, because if you have multiple components/systems in-line, you've got to test 'em to determine what's actually broke. This is what we noted were down:

• A couple of the office phones
• The FAX machine
• All networking except for one office, and it only for systems within that office
• The projector in the sanctuary
• The second monitor on the soundboard computer (connected with the projector using a VGA splitter)
• Part of the lights in the sanctuary

Once we knew what was down, we started looking at the component parts. For instance, not all the phones were down. So we took the ones that were and tried to connect them to the lines that were working. No dice. When we plugged them in to the power bricks that were working, they were dead. So we knew the phones themselves were history. With respect to networking, the DSL router was fried. It didn't even power up. We had another power cord that was known good with the same power characteristics and that didn't work. The DSL router was also a switch. So I grabbed a known good switch and tried to network the computers. No Internet, but access to printers and the like could be had... except that didn't work, either. I suspected the NICs on those computers, especially after one of the office computers didn't recognize its NIC any longer. However, another did recognize its NIC, but I couldn't get connectivity. I suspected the wiring, especially since I had one component I believe to be good not working when connected to the switch. In any case, you get the idea. We did this for all the components.

Have Your Insurance, Service Provider, and Hardware/Software Reselleres Contact Information Ready

Once we had determined there was some damage, one of the church staff placed a call to the insurance provider to verify the process for submitting a claim. Before we did anything with the equipment, we wanted to know how to proceed. We were able to get the information and start handling the equipment appropriately. The last thing we wanted to do was handle the equipment in such a way that the insurance company could say, "You violated subsection A of your agreement and we won't reimburse." We also contacted the DSL provider and had a new DSL router sent. Part of recovering from a disaster is filing the claims to be able to pay for equipment that has to be replaced. And part of it is understanding where you can get the replacement equipment.

Determine Work-Arounds

We had an action plan on how to proceed with recovering capabilities (we're still executing on it). In one case, the Internet connectivity, once we got the DSL router in, like most nowadays, it had wireless capability. Thankfully, though it looks like our phones/networking got hit by the lightning, once the new router came in, it was apparent that we still had DSL service. I was able to get that connected and running fairly quickly. However, since I'm almost positive the wiring is shot but I can't be sure about the NICs, we ran out and grabbed a couple of USB wireless NICs to install to provide some capability for Internet connectivity within the church. I knew there was a strong signal throughout the top floor because I did a walk around with a laptop and made sure of it. There are plenty of tools out there for this purpose. But a rather simple thing to do is walk around connected and just see if you can hit particular web sites. That's the low tech solution and it works just fine. Because I didn't have my normal security tools on this particular laptop, I went with the low tech option. The USB wireless NICs worked great and we had two systems on-line on the Internet.

Keep a Running List of Unresolved Issues and Reprioritize As Necessary

One of the applications we use for AV was built for two monitors. We were able to determine the second output of the video card had been damaged, which meant we were down to the single monitor. While the second monitor was good, we were able to verify the projector was hit, too. We had a spare projector, but no one had done the research on how to get the application running on a single monitor. We had temporarily used PowerPoint and hand-typed lyrics in, but we wanted to use the app because it handled this and handled it well. So after Sunday morning, we looked at where we were and noted that we still hadn't fully restored that capability. So before evening service we sat down and did that research. While running on a single monitor isn't ideal, it's do-able until we swap out the soundboard computer within the next week.

Determine What Can Be Done to Avoid the Disaster or Respond Better to It (Lessons Learned)

In our particular case, we had surge protectors and the like in place, but we're talking lightning here. Some of it was sufficient based on the exposure of the equipment, but some of it wasn't. We're looking at the building and seeing what else can be done, but there's only so much that can be done in this situation. That's one of the reasons it is smart to carry insurance. But if you have a failure, use it as an opportunity to learn. What could have been done to avoid the situation, if any? Is it reasonable for the business? What was done that could have been done better to recover? Were the recovery procedures satisfactory? Is there something else that needed to be done? Once you've looked at the situation, go back to your recovery plan and make the appropriate changes.