It's amazing how frequently the world stops for these "historical events"

I've been saying AI hasn't been tested properly. All the showcases I see aren't really pushing it far enough and they're always things you could in theory do by searching answers online. The real value is in all the problems that cannot be solved by online search.

Btw, this is why no one should think AI-ops is a thing yet. All the promises of powerful, hands-off AI operations, preventive fixes, etc. are just BS.

I’ve seen the chats about cascading failures and it’s impossible to not expect AI is involved somehow (likely brain drain and what not), but what are the options?

It sure looks like “fired everyone who knew what they were doing” but I feel like at this point you’re breaking open the check book and calling them, and even if they did that I’m assuming this shouldn’t be a “normal” event?

A bad antivirus pattern file can cause enormous CPU spikes. E.g.: a very inefficient regular expression with too much backtracking. This tends to get pushed out every few hours to every server, irrespective of role. Hence, "rolling back" the servers to a "known good image" doesn't help, because within a few minutes they get the pattern update anyway and die with CPU overload. (IIRC this type of issue caused a Cloudflare global outage recently.)

Similarly, someone here mentioned a very recent OpenSSL security update that also caused enormously high CPU load. These critical patches have been rolled out very quickly with minimal testing because without it, systems are very vulnerable to nation-states utilising AIs to rapidly exploit new vulnerabilities. Microsoft (probably) doesn't use OpenSSL much, but if they do, it would be everywhere because its use in layer-7 load balancers tends to be "pervasive" and hence a sweeping update can absolutely trash performance of everything in the data centre. (However, this would have been fixed by rolling back)

Most other explanations wouldn't account for the global outage.

This is big for medium/large fleets: fairly good central authentication, as well as being able to set policies for machine management (GPO).

I've also heard good things about PowerShell.

That is despite Microsoft ...you do know they tried to fire the guy who created it?

This has two assumptions:

1. people who knew what they were doing were all fired. maybe they just lost touch with the system reality and outsourced too much to the shaitan;

2. if the people were to be called back they can still understand what the system become after shaitan was let loose in it

You bring one system online, then the next, the next; you load balance it carefully, etc.

But in these catastrophic scenarios, those involved (AI particularly), want to bring everything up all at once and that itself causes some DDoS like attacks on internal and external systems. And that's assuming you solved whatever "unhappy" system took it down in the first place.

The thing I've see with using AI for the past 6 month is it's really too dumb about how long things take. Humans have a normal sense of how long a process will take, whether a race condition can exist (eg, 1ns vs 10s), and other time based details. Whether a log is produced before something happens or after, etc.

I've been considering adding time stamps to mine plugins to track the time, but it's sorta like you need to track it both at the start of the request, at the end of it, at the bewginning of the generation, at the end of the generation.

Anyway, AI has to be shit to use in these scenarios because it can't tell that a system takes 5 minutes to bring up, and if you bring up 10 of the same systems at the same time, they have zero capabilities against race conditoins on shared resources etc.

> make no mistakes and do it fast

part.

investigating..

goofing..

finagling..

hey claude what the hell you deleted all my configs permanently!!

metastasizing.. You’re absolutely right. I went above and beyond what you had asked and started completely fresh. Unfortunately I didn’t take any backups and deleted all existing distributed configurations. Oopsie!

Unless it causes enough economic damage for governments to interfere, nothing will change other than MSSharePrice increasing.

Call Sam Altman for some RAM first!

but I still can't send an email in Outlook

the hell is going on?

> Engineering teams are cycling affected infrastructure through traffic drain, restart, remediation, and staged return-to-service while monitoring closely.

so they have no clue what is going on...

The corrupted executives need to learn their lesson on consequences of their actions. Laying off the core of your system?. Say goodbye to triple-9 SLA.

I have zero sympathy for big tech

One can only hope for a damage to their reputation so significant, it encourages other companies to swtch providers, and the devs to stand up and develop competing solutions.

Do you think they actually care about their "triple-9 SLA"? Only in as much as they could use it as a marketing bullet point to sign contracts to increase revenues/profits.

It's been 18 hours since the start of the outage. If service isn't restored in the next 18 hours (36hrs+ outage), you can get the entire fee waived.

Microsoft gives a 99.9% monthly uptime guarantee, which means maximum allowed downtime is 7hrs.

The loss for a random due to MS software at this point is about 1400$ per seat (but probably at least double that) and rising.

Which is why every manager should now realize that their microsoft contract can never work as a valid disaster recovery option.

The response was basically, "we have a uptime garuntee and will pay out for downtime". But when it's clear _all_ of their clients have the same SaaS contract garuntee and a serious enough disruption would see _every_ client needing payment back, well, it's not much of a warranty is it.

Now I'm sure the SaaS has the redundancy in their system, but AI is the next deep integration that's going to hallow out a bunch of companies because some third party is just going to have a catastrophic loss somewhere down the line.

That might then become a problem when things stop working.

The meeting was obviously in teems, because they sold us on that already and most of the presentation the screen sharing didn't work for half the people, which generated a continued stream of office-grade jokes for the whole duration of the presentation (30 minutes or so).

As a guest the presenter was totally oblivious to this fact, as they didn't get access to the chat messages, because the policies are in place (we are a serious company).

[1]: https://nvie.com/posts/15-years-later/