Some Quick Lessons Learned From Massive Regional Power Outages

CTOvision, CTOlabs and the tech consultancy behind them Crucial Point LLC are all run out of the Washington DC area. We seek global views of technology but this region is definitely our base and we have a very high percentage of readers in the local ecosystem. For those of you here, you know the biggest tech story right now is not the latest software or hardware or tech concept or even our favorite Big Data topic but something that has impacted humanity from our very existence, the weather. The weather has had an absolutely huge impact on our tech these last few days.

Massive storms with a name most of us never heard of and probably never will again (derecho?) plunged over 3 million American’s into a life without electricity. For most that also meant a life with no or very poor cell coverage including data to mobile devices. Every day we read reports of more people getting their power back but it is shocking how many still are without. What is the impact on technology professionals? The fact is that even technology professionals are human and need certain things in order to function so a large part of the region’s technology workforce has other things on their mind right now so that impacts optimal use of technology in enterprises in the region. There are other issues to think through as well:

– Many buildings in the area have backup power driven by emergency power generators. But in most cases those generators are designed to operate for short outages. Longer outages will require them to be refueled and may cause them to come out of compliance with state and local regulations. These generators are highly regulated because of their impact on air quality.

– A surprising number of data centers had issues because of these storms. This included Amazon’s Northern VA facility. This was actually their second outage at this facility in one month. It impacted several businesses including many consumer-facing capabilities like Netflix and Pinterest. Most of us in the tech community still regard Amazon as the champion thought-leader that kicked off this whole modern approach to “Cloud Computing” with their new business model (for some history see Jeff Bezos’ Risky Bet from Businessweek Nov 2006). But increasingly we will have to ask ourselves, are they a a web service provider or a book seller? Which one is more important to them?

– Most all government organizations were impacted by the power outages, including DoD’s DISA, who, like many others, executed emergency plans that directed their workforce to make maximum use of telecommuting.

– Cybersecurity professionals sought to find lessons learned that could inform ongoing debate and discussion on cyber policies. For those professionals who already believe gloom and doom scenarios most seemed to take this as an example of what a major cyber attack could do. For those professionals who believe the threat of massive cyber infrastructure attack is over hyped they generally took this as proof that mother nature is far more of a threat to our infrastructure than nation-state attackers. I believe personally this is a test case that should be studied to improve infrastructure resiliency and lessons can be learned on how to bounce back from disaster faster.

We will capture other thoughts/considerations in the near term. Please let us know your thoughts. How did you do in the outages and what are your lessons learned?