If you can't take days to do it then do a gradual rollout in hours. It's not a high bar.
"Developers can use frameworks such as DriverKit and NetworkExtension to write USB and human interface drivers, endpoint security tools (like data loss prevention or other endpoint agents), and VPN and network tools, all without needing to write kexts. Third-party security agents should be used only if they take advantage of these APIs or have a robust road map to transition to them and away from kernel extensions."
Specifically the 2nd sentence above says security software should use the APIs, not Apple's kernel extensions.
I certainly don't want to know (through disaster news) about the construction company that built the bridge I drive through everyday, not for another 15 years, not ever!
This kind of software simply should not fail, with such a massive install base on so many sensitive industries. We're better than that, the software industry is starting to mature and there are simple and widely-known procedures that could have been used to prevent it.
I have no idea how CrowdStrike stock has only dropped 10% to the values of 2 months ago. Actually, if the financial troubles you get into are only these, take back what I said, software should be failing a lot (why spend money on robustness when you don't lose money on bugs?)
I actually expected their stock to drop a lot more than this, but goes to show you how valuable they are. investors know that any dip is only temporary because no one is getting rid of crowdstrike.
Think of the security landscape as early 90's new york city at night and crowdstrike as the big bulky guy with lots of guns who protects you for a fee, if he makes a mistakes and hurts you, you will be mad but in the end your need for protection does not suddenly go away and it was a one time mistake.
You're trying to hand-wave away the inexcusable. The outage is a symptom. The problem is the lack of even the most basic testing.
Clearly these files are sent out without even a minimal sanity check. That is a problem, and it's not something that can be hand-waved away.
This is inexcusable, point blank. “A couple of hours is a long time” is not a valid excuse when the alternative, as clearly evidenced, is millions of computers and critical systems simultaneously failing hard.
This might have been different if it was a small subset of computers, but this clearly could have been caught in minutes with any sort of sensible testing or canary rollout practices.
If problem encountered, don't send it out to everyone else.
They certainly run their software on those many customers' systems, but but based on my experience with them, "protect" isn't a descriptor I'm willing to grant them.
We don't have the counter-factual where Crowdstrike doesn't exist, but I'm not convinced that they've been a net economic or security benefit to the world over the span of their existence.
Sure, that applies to my company, but the counter-factual isn't "nothing is done and we keep getting pwned", the counter-factual is that instead of the resources spent on crowdstrike and their various problems (which have been regular since we adopted them, the recent mess was just the biggest), those resources are spent on improving security infrastructure without crowdstrike.
I can see your perspective, but you should consider this: They protect these many companies, industries and even countries at such a global scale and you haven't even heard of them in the last 15 years of their operation until this one outage.
You can't take days testing gradual roll outs for this type of content, because that's how long customers are left unprotected by that content. Although the root cause is on the channel files, I feel like the driver that processes them should have been able to handle the "logic bug" in question so we'll find out more over time I guess.
For example, with windows defender which runs on virtually all windows systems, the signature updates on billions of devices are pushed immediately (with exception to enterprise systems, but even then there is usually not much testing on signature files themselves, if at all). As far as the devops process Crowdstrike uses to test the channel files, I think it's best to leave commentary on that to actual insiders but these updates happen several times a day sometimes and get pushed to every Crowdstrike customer.