January 31, 2023
Every day, businesses face uncertainty and risks that are totally out of their control. Mitigating those risks are what responsible businesses should do, and that is what this guide will help you do.
In this article, we share our knowledge and experience with you as a disaster recovery solution vendor. We will share 4 of the most common scenarios that will require activating your disaster recovery plan. Then, we will teach you the basics on how to prepare and recover from these disasters should it ever happen to you. Towards the end, you will find additional resources and specifics on how to create a solid disaster recovery plan for 2023.
User Error – User error will continue to be the number one reason why businesses need to initiate a disaster recovery plan. User errors can range from simple mistakes such as accidental file deletion, typo, and password mismanagement to construction workers accidentally destroying cable or power lines. User errors should never be underestimated because they can have a colossal impact, such as a typo that took down Amazon for an afternoon or a bug that took down Facebook for 6 hours.
Cyber Threats – Cyber-attacks, typically in the form of ransomware, has exploded since 2020 as more and more office workers started working remote. According to Checkpoint, global attacks increased by 28%in the third quarter of 2022 compared to same period in 2021. Most cybersecurity experts agree that being attacked is not a matter of if, but a matter of when. The best approach to increase cyber resilience is through a layered approach with your disaster recovery solution as the last line of defense.
Power Outage – Power outages can cause significant disruptions to operations, and the business loss varies by industry. According to a survey by Consortium for Electric Infrastructure to Support a Digital Society (CEIDS), the average cost of a one-second outage among industrial and Digital Economy firms is $1,477, vs. an average cost of $2,107 for a three-minute outage and $7,795 for a one-hour outage. Severe weather during summer and winter will put additional strain onto the energy infrastructure, increasing the likelihood of power outages even in major metropolitan areas.
Natural Disasters – Natural disasters has caused $268 billion in damages in 2022, mainly driven by Hurricane Ian. These unexpected events often take businesses by surprise, which is why most businesses are unable to fully recover from hard-hit disasters.
Start with Business Continuity Planning – Before diving into disaster recovery planning, you need to start with business continuity planning. Although the two are similar in some respect, business continuity is broader and encompass the business as a whole. That is when business needs (such as acceptable level of downtime, RTO) and other requirements are uncovered.
Create a Plan with the Essential Elements – The disaster recovery plan, at the very least, should include the acceptable Recovery Time Objective (RTO) and Recovery Point Objective (RPO). In short, RTO is the acceptable downtime, and RPO is the acceptable amount of data loss. The IT department and the business as a whole should agree on both the RTO and RPO per business requirements (which should be uncovered in the business continuity planning phase). Further, the plan should include how often IT recovery should be tested.
Build a Checklist – Building a checklist lists of the specific required tasks when an outage occurs. At the very minimum, this list should be specific enough to guide general IT staff to initiate data recovery and restoration if the IT admins are not available (assuming the DR solution used is not “one-click” simple like what Quorum provides). At best, there should be checklists for both IT staff and management to follow in the case of an outage.
Define Activation Triggers – Knowing when to initiate a disaster recovery plan is just as important as what to do when a disaster occurs. We’ve seen companies initiate the whole bare metal recovery process after accidentally deleting a few files. We’ve also seen companies hesitant to activate disaster recovery even when they experience a major hardware failure, thinking they could’ve fixed the failure in a few minutes but instead took almost a day. There’s no right or wrong answer here. After testing the disaster recovery process, you should know how much resource is needed to initiate the disaster recovery process and what scenarios warrants triggering that process.
Build and Test DR Scenarios – An untested disaster recovery plan is almost as good as no plan at all. Many unforeseen obstacles and issues will arise as you go through the disaster recovery process. We’ve seen a lack of software update wreck a solid disaster recovery plan. We’ve also seen team miscommunication and technical glitches ruin the whole recovery process. Further, major disaster scenarios, such as the four mentioned above, should be tested because they are most likely to happen.
As we have learned with Met Opera’s ransomware attack, when developing a plan you need to include all potential actors. With Met Opera, the security team has stepped in and is driving all steps post-attack. We are not sure this was expected or part of the actual plan. If this is how the organization wants to run things, that’s fine, but it should be known in advance and be part of the test scenarios.
Before the actual planning takes place, you need to assess your organization’s needs, resources, and goals. That is what disaster recovery PRE-planning is all about. This includes business continuity planning (as mentioned earlier), defining people’s roles and responsivities, and much more. Further, once the disaster recovery process is initiated, there’s a list of management tasks that requires completion, including starting a Call Tree, deciding on command center, and more. We have compiled a list of pre-planning checklist and a list of management tasks in this recorded webinar here for those of you who are serious about mitigating risks and being prepared.