So, even when you fix the issue, it takes a bit to clear up, and the load gauge stays at the red zone for a while.
This lets you start at a very small amount of load and gradually ramp up. But you can’t reject randomly or goodput stays at zero and the backlog never decreases.
I believe SDN systems can do that with ease, but what would be the equivalent method on older systems? Selectively allowing IPs to go through the trickle load into the system?
Distribution of users by /8 isn't very even and the order that customers will get online wouldn't be an order anyone would pick, but it should be reasonable to implement and synchronize (if needed) and it lets you make progress.
Another idea is to have different clients use different ports (enterprise customers on one port, SMB on another) and turn up in the order of SLA.
I've definitely been in similar scale war rooms and 15 hours is not acceptable.
The thing with plain outages vs. say, partial degradation of services, you can roll out fast because you're already at worst case scenario *[1].
Here's an interesting one to compare to that I'm very familiar with:
https://status.cloud.google.com/incidents/xVSEV3kVaJBmS7SZbn...
Another really bad one I did not have any involvement in but studied extensively post hoc:
https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1S...
*[1] - I suppose data loss is worse but let's assume most services are stateless.
What's not acceptable to me is a bug of this impact going undetected for 6+ months in production.
You can't even say "but managing fleet/cells is harder": they make software for that, too!
- our testing didn't cover it (again, this is expected occasionally).
- this cascaded to take down more than a billion people's service (WTF; this is not how you design systems).
- it took 15 hours to deploy the fix (not good, but at this scale maybe).
Feels like they are completely missing the point of where the problem is and how to fix it?