Leveraging Disaster Recovery for Business Continuity

I live in Florida and I’ve always had to live with hurricanes and their effect on my personal life as well as my business life. I’m writing this on Friday, 9/30/2022, just after Hurricane Ian ravaged the West Coast of Florida. I live in Southeast Florida on the Treasure Coast and I have family (Aunt, Uncle, and Cousins) that live in Port Charlotte, FL that was directly impacted by Hurricane Ian.

There are parallels between the things that people must do to personally prepare for a hurricane and what must be done to prepare businesses. There is an absolute mountain of information (some of it conflicting) on how to prepare for a hurricane. Similarly, there is also a plethora of information on how to prepare and strategize to create a Disaster Recovery or Business Continuity plan. The goal of this blog is to discuss the differences between Disaster Recovery and Business Continuity plans and to provide guidance on how to create each. There are several types of disasters that should be considered when creating one of these plans. Due to my recent experience with Hurricane Ian, I’ll be focusing through the lens of hurricane preparation. Although, as I learned by assisting several organizations through the Covid epidemic - the principles that are used for hurricane preparedness easily translate to other disasters, like Covid.

The primary goal of a Business Continuity plan is to keep your business running during a disaster. There are easily identifiable organizations that should focus on keeping everything running during any type of disaster. Any organization that is involved in public health or safety should try to put together a Business Continuity Plan. Hospitals, local news outlets, police, fire, and Red Cross operations would be a few examples.

The primary goal of a Disaster Recovery plan is to get your operations back up as soon as possible after the disaster abates. During a hurricane, it doesn’t make sense to keep a gas station or grocery store open since conditions will make travel extremely dangerous and oftentimes impossible. For these businesses, it would seem that the prudent thing to do would be to wait for the hurricane to pass and then reopen for the community as soon as possible after the storm passes.

Not all businesses fall neatly into one category or the other. For some, the need ends up focused on business divisions (think sales, engineering, operations, finance, recruiting, and repair). For others, the focus is on applications (inventory, point of sale, customer relationship management, electronic records, repair ticketing, dispatch, and communications). For some businesses, they have to prioritize by application and decide which applications need Business Continuity, and which applications need Disaster Recovery.

You can break down the process into phases so that it’s less intimidating. In my experience, no two businesses have the exact same plan, but the process is generally the same for all. Here are the main phases that typically go into disaster preparation:

1. Strategy

2. Planning

3. Documentation

4. Testing

5. Repeat 

For this post, I’ll use a hospital as an example. This is intended to be an example of some of the thoughts. It is in no way meant to be a comprehensive plan - just an example of a high level thought process.

1) The Strategy Phase

The key activities during this phase are to identify, prioritize, analyze, and assess your business. It’s helpful to break down the business into functional pieces. Prioritize the functions into those that are critical, important, and non-critical. Critical business functions typically correspond to Business Continuity while important business functions map to Disaster Recovery. In some cases, you might find that all of your functions fall into only one category. Next you’ll need to analyze each function and look for points of failure. For each point of failure, you’ll need to assess the corresponding impact from failure. 

Here are a few hospital functions that might be considered and an example of how they might be viewed - this is not meant to be a roadmap and I figure some might disagree with the categorizations. This is just an academic example:

Critical:

• Emergency room

• Housekeeping

• Cafeteria

• Electronic Medical Records

Important:

• Billing

• Human Resources

Non-Critical:

• Grounds keeping

• Elective surgery

• Community Outreach

Here’s how someone might analyze medical records:

1. The hospital uses a cloud hosted software company for patient electronic records.

2. Records are accessible via VPN over the internet.

3. The hospital only has one internet connection.

4. If the connection is lost, there is no way to access critical patient information.

5. *** Critical Business Need to ensure that internet connectivity remains intact during the disaster so that electronic records are available. Follow the Business Continuity path.***

Here’s how someone might analyze Human Resources:

1. Employee files are stored on a local server.

2. The local human resources server is securely backed up to the cloud.

3. Backups happen nightly starting at 3AM.

4. If the local server is compromised, it can be restored from the cloud backup.

5. *** Important - need to ensure that employee files are available and accessible, but not critical during a disaster. Follow Disaster Recovery.***

2) The Planning Phase.

Using the examples of Electronic Medical Records (or EMR for short) and Human Resources (or HR for short), here’s how someone might start to plan:

EMR:

• Vulnerability: Single threaded internet connectivity.

• Solution: Find secondary and tertiary internet connections for diversity and redundancy.

• Plan:

  • Delegate project to IT Procurement to find and procure the additional internet connections.

  • Delegate project to IT Network Engineering to connect additional internet connections to existing infrastructure.

  • Allot 120 days or sooner to accomplish this task.

HR:

• Vulnerability:

  • The server that houses HR records is physically on-premise and could suffer equipment failure. 

  • In the event of equipment failure, all HR data between 3am and the time of failure would be lost.

  • HR would not have access to their files until a suitable replacement server is procured and installed.

• Solution: Ensure that the backup copy of HR files is accessible during an equipment failure onsite.

• Plan: Delegate project to IT Network Engineering to contact DR company and change the service plan to include file access during a local server failure.

For some additional information on Disaster Recovery or Disaster Recovery as a Service (DRaaS) vendors, check out our DRaaS guide and our post on factors to consider when buying DRaaS.

3) Documentation phase

I’ve always found this phase to be the most straightforward, but oftentimes it’s very poorly done. The hospital should keep records for all of its critical components.

EMR:

• EMR Provider:

• EMR Provider contact information:

• Relevant account information to provide to EMR for assistance:

• Internet Service Provider (or ISP for short):

• ISP Contact Information:

• Relevant account information to provide to ISP for assistance:

• Internal network diagrams for:

  • Internet Equipment (Firewalls, Routers, Switches, WiFi access points)

  • Passwords

  • IP addresses

• In the event of a circuit failure, how the secondary and tertiary circuits used to access EMR.

HR Files:

• Disaster Recovery as a Service (or DRaaS for short) provider:

• DRaaS Provider contact information:

• Relevant account information to provide to DRaaS for assistance:

• In the event of a server failure, how and when to contact the DRaaS provider and ask them to make the HR Files live. How to access the HR files on the DRaaS platform.

4) Testing Phase

Now that you’ve set up your Business Continuity plan for EMR and the Disaster Recovery plan for HR Files, it’s time to test the plan to see if it works.

EMR:

• Fail the primary circuit, secondary circuit, and tertiary circuit:

  • What was the expected result?

  • What was the actual result?

  • Is this a satisfactory outcome?

  • Do the configurations need to be fixed?

HR Files:

• Take the local HR File Server offline.

  • Contact the DRaaS Provider and ask them to go through their process to make the backup of the HR Files live and accessible.

  • Wait for confirmation from DRaaS provider that their process is complete.

  • Have local HR Staff access HR Files and ensure they’re able to see what they’re supposed to see.

  • Did it work?

  • Is this a satisfactory outcome?

  • Does the process need to be revisited?

5) Repeat Phase

It’s extremely likely that new functions are regularly introduced into the business. Every business should set up a recurring interval to go through this process. The interval should make sense to everyone involved. In the case of our hypothetical hospital, a standard interval would be every 6 months to go through all these phases.

If, after reading this, you still feel overwhelmed or if you’d just like to have assistance with all of this, please reach out to Lightyear. We have sales and engineering professionals on staff that would be happy to help you with this process.

My Personal Conclusion!

As a follow up. It’s now Sunday 10/2/2022. We did hear from our family in Port Charlotte yesterday. They were doing well. Their house took very minor damage and they were getting by using a small generator to power their critical appliances, like the refrigerator and ceiling fans (Business Continuity). The central Air Conditioning and TVs were set aside for the time being, but they’re looking forward to getting them back online as soon as possible (Disaster Recovery). They were, however, unable to procure gasoline on the West Coast for a number of reasons - chief among them was that the roads were not safe for the large fuel tankers to come through yet. We decided to go help them out. Here are a few pictures of our adventure:

My son and I loaded up the truck with our gas cans. 

We drove through hazardous debris, downed power lines, and washed out roads:

We were proud to see our Florida National Guard taking part in the community’s recovery process:

We were very relieved to find our family members either lounging by the pool (where there used to be a nice screen enclosure) or decorating for Halloween. When we got there they were out of gas for their generator and very happy to see us.

As a side note, our family was very fortunate to have only taken minor damage during Hurricane Ian. The storm’s impact on Fort Myers, the surrounding areas, and through the central part of the state was immense. If you’re able to - I encourage you to find a way to help or donate to the people that now have a massive job of rebuilding their community.