Re:AWS us-east-1 outage
by fragmede on Tuesday December 07, 2021 @11:19PM
(#29481206)
The tactical point is to remove sharp edges, eg there's a tool that optionally take a region argument.
network_cli remove_routes [--region us-east-1]
Blaming the operator that they should have known that running
network_cli remove_routes
will take down all regions because the region wasn't specified is the kind of thing as to what's being called out here.
All of the tools need to not default to breaking the world. That is the first and foremost thing being pushed. If an engineer is remotely afraid to come forwards (beyond self-shame/judgement) after an incident, and say "hey, I accidentally did this thing", then the situation will never get any better.
That doesn't mean that engineers don't have the ability to break things, but it means it's harder (and very intentionally so) for a stressed out human operator to do the wrong thing by accident. Accidents happen. Do you just plan on never getting into a car accident, or do you wear a seat belt?
All of the tools need to not default to breaking the world. That is the first and foremost thing being pushed. If an engineer is remotely afraid to come forwards (beyond self-shame/judgement) after an incident, and say "hey, I accidentally did this thing", then the situation will never get any better.
That doesn't mean that engineers don't have the ability to break things, but it means it's harder (and very intentionally so) for a stressed out human operator to do the wrong thing by accident. Accidents happen. Do you just plan on never getting into a car accident, or do you wear a seat belt?