Surviving the Storm:
How your disaster recovery plan will determine if you sink or swim
According to reports:
- More than 50% of businesses will permanently shut down as a result of a significant data loss related to a natural disaster.
- Downtime will cost companies between $50,000 and $5,000,000 per event.
- More than 60% of businesses don’t have a documented disaster recovery plan and 40% of the companies that had plans actually failed in the recovery process.
In the technology sector, we are witness to hundreds of unfortunate situations that result in outcomes that can cripple the future of an organization. We know how important it is to have the right plans and the right solutions in place, and how important they are to the survival of a business. Nonetheless, when we put plans in place, we still hope we’ll never actually have to execute on a full business disaster recovery plan. On the night of March 13th, that all changed. The flash floods resulted in rising water which penetrated our buildings. We were helpless. Mother nature had reminded us who was in control.
While it is impossible for any plan to be perfect, we can speak from firsthand experience that a very thorough plan is mandatory for any organization to have a chance in being able to operate in an emergency situation. In our circumstance, we executed our plan at 9:07 PM and continued taking calls and supporting our clients, proudly we didn’t miss a single call or service request. There were many things that transpired throughout the evening that we did right and some things we can improve.
Beyond having a robust and documented plan (ours is 172 pages and growing…), the primary catalyst to a business recovery plan is communication. We have all come to rely on our mobile devices as a primary method for communicating. In our situation, we were fortunate that we didn’t lose cellular services, but we have witnessed disasters in Joplin and in Norman, Oklahoma where cellular services were either completely down or severely impacted. One of the first major components of our plan outlines all communication paths that may be necessary. This includes primary and alternate phone numbers, primary and secondary email addresses, emergency contacts and specific roles.
In addition to communication with your organization, getting communications to your customers and vendors is also very important. Our recovery plan has an outline of all key contacts to ensure we don’t have to search during a disaster. This includes our supplier vendors, utility and communication contacts, insurance contacts and policy numbers, building contacts and more. Also, as soon as reasonable, it’s important to get a message to your customers to let them know what is actually occurring. In all circumstances, be sure to be transparent. If you are going to be down, be honest about the status of services or products. If you are going to be in business, but with delays, share exactly that. Provide updates as appropriate throughout the process to ensure there is no misperceptions about what is going on. The last thing you want are rumors driving perceptions.
Like all good plans or initiatives, there must be a responsible party (or parties). The responsible party should know the situation and be able to put the appropriate components of the plan into action. As the CEO, I often feel like I should be the person in charge in a situation like this. The reality that I learned is that I was dealing with too many immediate challenges to coordinate communications and plans for the next day. Having someone else responsible for the plan enabled myself and others the ability to focus on those immediate challenges. While I was dealing with people, emotions, emergency response professionals, and more, they were focused on communications, phone systems migrations, anticipation of what may or not be possible the next day and responding accordingly. This was critical to success for the next day.
Offsite resources are mandatory for both the technology and helpful for the people aspect of a successful recovery. In JMARK’s situation, we leverage an offsite datacenter facility to house many of our primary systems and connectivity. The degree in which your organization must sustain operations will determine the design of your infrastructure. If you must operate 24×7, then utilizing a datacenter with redundancies is a requirement. If you can tolerate a moderate amount of down time, offsite backups may be sufficient. However, your plan must include arrangements for your recovery mechanisms. This may include a standby system at a secondary location or the means to restore servers and workstations in a temporary cloud or secondary facility. There is obviously a cost analysis that should help drive these decisions.
Proper insurance coverage
Having the proper level of insurance coverage may seem like a given, but the only reason that JMARK had the proper coverage in place was because of the review we do with Ollis/Akers/Arney on an annual basis. This included the proper types of coverage at the proper levels for our current business. Having this in place truly provided piece of mind while we were dealing with the challenges of the night and the following week. We could move forward without concern that we were going to be able to get back on our feet.
Team & Culture
One of the most important mantras at JMARK, is “people first, technology second.” I can’t emphasis enough the significance of having the right team and culture to deal with adversity. There were many people within our organization that were up all night, dealing with many unpleasant areas, handling planning, communications, moving systems and dozens of other priorities. Without a true team of professionals that were focused on Jmark’s commitment and success, that evening would have been a disaster in more than one way.
On the note of people, training around the plan is required for the plan to be successful. Taking the time to share the details of the plan with the appropriate people in the organization is the only way the plan has value. In our case, we were fortunate because we had recently completed the internal review of our plan and determined how to store and distribute the plan in the various emergency scenarios.
Technology planning & recovery
Identifying the key components of technology, specific to your organization; and determining the available means of sustainability, recovery and the priority for each application is one of the fundamental portions of a healthy plan. Each of the applications used to run the organization should be prioritized and evaluated for the necessary uptime or recovery time. As an example, the order processing system would be critical, a Voice-Over-IP phone system is critical, but in most circumstances a system that only does accounts payable can be offline for a day or two. The priority management list will determine how to design the technology solution and in what order to recover each component. This is very helpful for an external IT Management company to fully understand the requirements of the business.
Cloud solutions have dominated the technology news sector recently. Many of these solutions have actually been around for many years. However, in the last few years the cloud enabling technology has matured and the cost of communications have become much more reasonable. It is because of this that almost all organizations can now justify the expense associated with an offsite solution of some type. Whether that is a secondary office, a redundant cloud based solution or fully operating from a data center, there is an option available to all organizations. Taking the time to evaluate your entire infrastructure, prioritize, deploy and test the technical design is the only way to ensure that your organization doesn’t end up on the wrong side of the statistic of those businesses who succeed or fail.
There are obviously many more items that should be considered and discussed for a business recovery plan to be complete. I hope you never have to deal with a disaster, but if you do, I hope you have a comprehensive plan to follow.
Thomas H. Douglas
JMARK Business Solutions, Inc.
 Disaster Recovery Preparedness Council’s 2014 Annual Report