A bad antivirus pattern file can cause enormous CPU spikes. E.g.: a very inefficient regular expression with too much backtracking. This tends to get pushed out every few hours to every server, irrespective of role. Hence, "rolling back" the servers to a "known good image" doesn't help, because within a few minutes they get the pattern update anyway and die with CPU overload. (IIRC this type of issue caused a Cloudflare global outage recently.)
Similarly, someone here mentioned a very recent OpenSSL security update that also caused enormously high CPU load. These critical patches have been rolled out very quickly with minimal testing because without it, systems are very vulnerable to nation-states utilising AIs to rapidly exploit new vulnerabilities. Microsoft (probably) doesn't use OpenSSL much, but if they do, it would be everywhere because its use in layer-7 load balancers tends to be "pervasive" and hence a sweeping update can absolutely trash performance of everything in the data centre. (However, this would have been fixed by rolling back)
Most other explanations wouldn't account for the global outage.
This is big for medium/large fleets: fairly good central authentication, as well as being able to set policies for machine management (GPO).
I've also heard good things about PowerShell.
That is despite Microsoft ...you do know they tried to fire the guy who created it?
This has two assumptions:
1. people who knew what they were doing were all fired. maybe they just lost touch with the system reality and outsourced too much to the shaitan;
2. if the people were to be called back they can still understand what the system become after shaitan was let loose in it
You bring one system online, then the next, the next; you load balance it carefully, etc.
But in these catastrophic scenarios, those involved (AI particularly), want to bring everything up all at once and that itself causes some DDoS like attacks on internal and external systems. And that's assuming you solved whatever "unhappy" system took it down in the first place.
The thing I've see with using AI for the past 6 month is it's really too dumb about how long things take. Humans have a normal sense of how long a process will take, whether a race condition can exist (eg, 1ns vs 10s), and other time based details. Whether a log is produced before something happens or after, etc.
I've been considering adding time stamps to mine plugins to track the time, but it's sorta like you need to track it both at the start of the request, at the end of it, at the bewginning of the generation, at the end of the generation.
Anyway, AI has to be shit to use in these scenarios because it can't tell that a system takes 5 minutes to bring up, and if you bring up 10 of the same systems at the same time, they have zero capabilities against race conditoins on shared resources etc.
I’ve seen the chats about cascading failures and it’s impossible to not expect AI is involved somehow (likely brain drain and what not), but what are the options?
It sure looks like “fired everyone who knew what they were doing” but I feel like at this point you’re breaking open the check book and calling them, and even if they did that I’m assuming this shouldn’t be a “normal” event?