Me: "How do I connect [X] to [Y] using [Z]?"

Copilot: "Please select the AKS cluster you'd like to delete"

Can't say I've experienced many bugs in there either. It definitely is overpriced but I assume they all are?

At a large scale, azure is dramatically worse then Aws.

Don't forget extremely insecure. There is a quarterly critical cross-tenant CVE with trivial exploitation for them, and it has been like that for years.

But what we do when things are easy is not who we are. That's a fiction. It's how we show up when we are in the shit that matters. It's discipline that tells you to voluntarily go into all of the multi-tenant mitigations instead of waiting for your boss to notice and move the goalposts you should have moved on your own.

16:04 UTC on 29 October 2025 – Investigation commenced following monitoring alerts being triggered. ` A 19-minute delay in alert is a joke.

It would be nice though if alert systems made it easy to wire up CD to turn down sensitivity during observed actions. Sort of like how the immune system turns down a bit while you're eating.

I think if you really wanted to do on call right to avoid gaps you’d want no more than 6 hours on primary per day per shift, and you want six, not four, shifts per day. So you’re only alone for four hours in the middle of your shift and have plenty of time to hand off.

The reason is probably because changes to the status page require executive approval, because false positives could lead to bad publicity, and potentially having to reimburse customers for failing to meet SLAs.

We should be lucky MSFT is so consistent!

Hug ops to the Azure team, since management is shredding up talent over there.

You don’t want to debug stuff with low sugar.

    16:04 Started running around screaming
    16:15 Sat down & looked at logs

Very circular way of saying “the validator didn’t do its job”. This is AFAICT a pretty fundamental root cause of the issue.

It’s never good enough to have a validator check the content and hope that finds all the issues. Validators are great and can speed a lot of things up. But because they are independent code paths they will always miss something. For critical services you have to assume the validator will be wrong, and be prepared to contain the damage WHEN it is wrong.

Looks like there was no monitoring and no alerts.

Which is kinda weird.

I think it's perhaps a gap in the tools. We apply the same alert criteria at 2 am that we do while someone is actively running deployment or admin tasks and there's a subset that should stay the same, like request failure rate, and others that should be tuned down, like overall error rate and median response times.

And it means one thing if the failure rate for one machine is 90% and something else if the cluster failure rate is 5%, but if you've only got 18 boxes it's hard to discern the difference. And which is the higher priority error may change from one project to another.

Troubleshooting has completed

Troubleshooting was unable to automatically fix all of the issues found. You can find more details below.

>> We initiated the deployment of our ‘last known good’ configuration.

System Restore can help fix problems that might be making your computer run slowly or stop responding.

System Restore does not affect any of your documents, pictures, or other personal data. Recently installed programs and drivers might be uninstalled.

Confirm your restore point

Your computer will be restored to the state it was in before the event in the Description field below.