Extreme Redundancy – Don’t Leave Home Without It!

Visa’s command and control center—somewhere on the east coast of the United States—is a case study in redundancy with intense security, backups (for everything), and failover processes for system failures. And while most businesses don’t require these same levels of high availability for their own information networks and data centers, Visa can certainly teach a course on risk management to those who don’t believe such redundancy is necessary.

Visa’s command and control center—somewhere on the east coast of the United States—is a case study in redundancy with intense security, backups (for everything), and failover processes for system failures. And while most businesses don’t require these same levels of high availability for their own information networks and data centers, Visa can certainly teach a course on risk management to those who don’t believe such redundancy is necessary.

In the past four years, the unthinkable is happening more often than predicted. From the global stock market meltdown of 2008, to the Japanese tsunami disaster at Fukushima, to flooding in Thailand, and many other prominent examples, there’s Black Swans aplenty without signs of abatement.

Most of these events were near impossible to predict, but it’s entirely within the realm of risk management to prepare for heavy tails. The key to countering effects from extreme outlier events, says Black Swan author Nassim Taleb, is to build redundancy into business processes. He counsels companies, enterprises and even countries to “avoid optimization (and) learn to love redundancy.”

And while it may be a progressive example, Visa provides a well documented case study into managing fourth quadrant risk.  A Fast Company article titled, “Visa is Ready for Anything” provides a rare view in Visa’s data centers where 150 million daily transactions are processed.  Taking a page from NASA’s Mission Control, this data center monitors Visa network security, availability and capacity on a 24x7x265 basis. And meeting the definition of a “Tier 4” data center; “every major system—mainframes, air conditioners batteries etc, has a backup.”  With backups for backups and disaster planning in place, this data center is designed to withstand attacks from all sorts of events – terrorism, hackers, and even natural disasters such as earthquake and/or tornado.

While it’s true that Black Swans are about “unknown, unknowns”, and preparation for every type of risk is not only unrealistic but also too expensive, there is much companies can do to build more redundancy into daily operations. Redundancy is especially relevant as “interlocking fragility” of global communication, supply chain, and financial networks means correlations move to one with increasing frequency.

Today’s business culture celebrates “optimization” for every process; in essence living as near the edge as possible without falling off the cliff.  That’s a dangerous strategy for today’s interconnected world, especially because extreme events can take networks offline for hours, days and even weeks, costing companies tens of millions in lost revenues.

Critics of redundancy argue that robustness is too expensive. But Dr. Taleb counters; “Redundancy is like (being) long on an option, you certainly pay for it but it may be necessary for survival.”