Re:Summary of the Amazon S3 Service Disruption
by colanderman on Thursday March 02, 2017 @07:44PM
(#13779826)
For many systems, the rule is simply "X of Y servers must be up". Something like that isn't too hard to enforce.
In practice, it would likely be very hard to make a model of your infrastructure to test against, but I can imagine a tool that would run each query against a set of heuristics, and if any flags pop up, it would make you jump through some hoops to confirm. Such a tool should NEVER have an option to silently confirm, and the only way to adjust a heuristic if it becomes invalid should be formally getting someone from an appropriate department to change it and sign off on it.
By the way, this is how companies acquire red tape. It's like scar tissue.