The last few days I’ve been reading through Nassim Nicholas Taleb’s book The Black Swan: The Impact of the Highly Improbable. It’s an absolutely fascinating read, but I bring it up in this post because the lessons of his book have a very clear application to the area of disaster recovery.
Taleb’s basic premise is that we live in a world that is increasingly impacted by highly improbable but earth shattering events which he calls Black Swans. These Blacks Swans are innovations and catastrophes that few see coming but that ultimately bring about irresistable change–things like the rise of the Internet, or the 9/11 terrorist attacks. The problem, as Taleb sees it, is that we discount the power of these kinds of events and so we both place ourselves in dangerous positions where we are vulnerable to negative Black Swans and refuse to take the small risks that could set us up to harness the impact of positive Black Swans.
Disaster recovery ties in because, in many ways, it’s an attempt to deal with the kind of unpredictable events that can impact an organizations physical infrastructure. Essentially practitioners of disaster recovery strategies take for granted that disasters will occur and then try to prepare their organization to respond accordingly.
The problem is, that disaster recovery plans often don’t take into account the possibility of Black Swans. The drafters of these plans may acknowledge that they cannot predict when a disaster will occur, but they often act as if they can predict how. These plans read like detailed scripts, telling employees exactly what to do and when to do it. Now this in itself is not a problem. A clear set of disaster response instructions is likely to be invaluable when a disaster plays out like you predicted. The issue is that if your disaster recovery plan is nothing more than a set of these scripts, then you’ll be in big trouble when a disaster doesn’t go according to plan, that is when you face a Black Swan style disaster.
A Black Swan style disaster is essentially any disaster that you cannot imagine until after it has happened. It may be a completely novel catastrophe–the 9/11 attacks–a twist on a traditional disaster–an extraordinary powerful hurricane that floods a whole city–or a perfect storm of smaller disasters–a backup server failure the same day as an office fire and a hacker intrusion. Ultimately the point is that no matter how many disaster scenarios you imagine, and no matter how many detailed scripts you base off of them, there will always be something never expected.
So how do you deal with these Black Swan disasters? By fighting bad luck, with good luck. By setting your firm up so that in the midst of a disaster there is a chance for you to reap the benefits of a positive Black Swan. Essentially, any positive response your organization takes to a unplanned disaster is going to be the result of some employee being in the right place at the right time and taking the right actions. There’s no way you can have any control over the first two parts, but you can take steps to ensure that if someone in your organization finds themselves in the right place at the right time he or she will know what to do. Here are some ways to increase your chances of finding good fortune amidst bad
1. Set up redundant communication channels - In a disaster it’s likely that at least one of your normal communications channels will fail. Your email servers might go down, cell service might suffer, and as this year’s undersea cable disaster showed even the Internet is not invulnerable. You need to set up as many communications channels as possible so that your organization can still talk throughout a disaster. Here’s where technology can really lend a hand. Emergency notificaiton systems (we talked about these in a earlier post, Talking through Disaster) can blast out text messages to a large number of people very quickly. Many SaaS providers (Engage included) also offer failover solutions where they will run your communications systems on their equipment in a disaster situation. With enough redundency you’ll have a good chance of staying in touch during even the worst of disasters. That way employees can still tell management what’s happening on the ground and management can still pass employees the instructions they need.
2. Explain your disaster response goals to everyone involved – You can treat your employees one of two ways when it comes to disaster recovery. You can either treat them like robots, meant to be programmed to perform certain tasks with no emphasis on understanding, or you can treat them like free-thinking human beings with the power to improvise. The human approach is better because it gives you a better chance of surviving the black swan style disasters that lie outside your detailed recovery plans. If your employees are robots with no understanding of your firms ultimate goals, then they’ll be paralyzed when faced with the unexpected. They won’t be able to act because they will be unsure of what they are supposed to be acting towards. Even if they can overcome their uncertainty, there’s no guarantee they’ll actually take action that will help in the recovery process because they don’t know the big picture. But if employee’s are given a comprehensive view of the recovery process and it’s goals, then there’s a chance that they’ll be able to improvise in unexpected situations, helping minimize your losses.
3. Encourage heroism - Sometimes doing the right thing in a disaster is means going above and beyond the call of duty. You need to make sure that employees will be prepared to make those sorts of sacrifices if the time comes. That means not only rewarding above-standard contributions after the fact, but also making it clear these contributions will be rewarded. In discussions about disaster recovery companies should emphasis the importance of rising to the challenges that disasters might bring.
In fact, in many ways these strategies are already general best practices outside of the area of disaster recovery. Firms who encourage communication, give employees a big picture view, and create a culture where employees go the extra mile are going to succeed. They just need to ensure that those some practices are extended to the sphere of disaster recovery.
Updated: Since writing this post we’ve come across some other folks talking about the same things. Check out this Computer World article focusing on the people aspect of disaster recovery.
No related posts.
