Cascading failures, the downside of "Eat your own dogfood" [telecom]

Posted by brtech December 8, 2020

You may have heard that there was a pretty substantial outage in Amazon AWS Cloud services November 25, 2020. The summary of what happened can be found at:

and is emblematic of how a simple maintenance action can lead to layer upon layer of unanticipated effects. Amazon is an amazing company, the AWS service is really impressive, and what happened in this case could easily have occurred in any cloud service. I'm unaware of whether there was any direct impact of this particular outage on any NG9-1-1 service, and I'd be very interested in hearing if there was. The lessons to be learned here apply much more broadly than cloud services.