The easiest, and the default choice when I worked at Google, was exactly what you said: a sliding scale.
You could target racks, data centers, clusters, regions, etc. And “drain” the unit. For instance, stop routing traffic to cluster foo.
Then you could increment up stepwise 10%, 20%, etc.
Obviously we didn’t use nginx, and sure there’s interesting engineering involved. But it’s not like these systems are magic. There’s just a lot of clusters and a lot of load balancers. But at the end of the day, it’s just a load balancer.
As another Xoogler in this thread pointed out, there’s two problems:
1) Sometime teams accidentally introduce circular dependencies. Service foo might require service bar to boot and vice versa. Typically that’s fine, but from a rare cold start where both foo and bar are down, you’re in a pickle.
2) Load balancing is “easy”. But the services need to be designed with the aforementioned load balancing strategy in mind. Stuff gets tricky if there’s any pollers, process affinity, multiple components that need to process parts of the same user’s request in tandem. Given that outlook is ancient, I would not be shocked to learn that the system doesn’t respond well to naive “slowest ramp up traffic to a cluster” approaches. For big distributed applications, it takes a lot of meticulous design to build a system that responds well to simple SRE techniques like that.
That is one thing my team at Google would actually consider. We would make the code and architecture more complicated to accommodate the types of naive mitigations one would want to apply in an outage.
Extent of my knowledge is nginx or traefik load balancing which at this scale is virtually equal to no knowledge. But seems like there would be systems in place to simply drop a sliding percentage of excessive traffic or block some regions once critical thresholds are reached rather than compounding the problem with forwarding the growing number of requests downstream to next victim server.