News for hackers. Stuff that matters.
Re:Post Mortem of Google Outage on 14 December 2020 by jiggawatts on Friday December 18, 2020 @11:13PM (#25474953)
There are some "fun" horror stories of people hitting the Azure Resource Manager API quota limits. At least one of them was linked to some sort of shared service principal for the client org, so if anyone hit it, the whole organisation could get locked out.

It would first start to throw HTTP 429 codes occasionally, then you would get locked out for exponentially increasing times, up to two weeks or something absurd like that.

Even if you called support, there was not a lot they could do, because the rate limiter is a low-level thing built into an internal load balancer somewhere, and it is difficult to override it for a single account.

The worst part was that you could hit the limit while not actually taking any action! If you just had certain Azure Portal screens open, the JavaScript would refresh things in the background, constantly consuming the API rate limit quota. Some screens do so many calls that you'll hit the rate limit in a matter of minutes if you leave your browser on those. If you fail to close all browser sessions of all admins right away, you can lock yourself out for days.

Re:Post Mortem of Google Outage on 14 December 2020 by nullsense on Saturday December 19, 2020 @12:46AM (#25475397)
Yikes. That sounds nightmarish.
Re:Post Mortem of Google Outage on 14 December 2020 by dharmab on Saturday December 19, 2020 @01:06PM (#25479540)
My team struggled with this for two years. We ended up helping MSFT add a caching layer to the Kubernetes Azure cloud controller and creating a separate Azure subscription for each Kubernetes cluster.