Except the biggest IT outage ever. And a postmortem showing their validation checks were insufficient. And a rollout process that did not stage at all, just rawdogged straight to global prod. And no lab where the new code was actually installed and run prior to global rawdogging.
I'd say there's smoke, and numerous accounts of fire, which this can be taken in the context of.
Sometimes people are up in arms "where's my next version" (eg when adaptive headlights was introduced), yet Tesla prioritise a safe, slow roll out. Sometimes the updates fail (and get resolved individually), but never on a global scale. (None experienced myself, as a TM3 owner on the "advanced" update preference).
I understand the premise of Crowdstrike's model is to have up to date protection everywhere but clearly they didn't think this through enough times, if at all.
When I read the same release notes so often I begin to question whether this redesign is really taking more than six months to roll out. And then I read the Sonos app disaster and I thought that was the other extreme.
Google is terrible at release notes. Since several years ago, the release notes for the "Google" app on the Android app store always shows the exact same four unchanging entries, loosely translating from Portuguese: "enhanced search page appearance", "new doodles designed for app experience", "offline voice actions (play music, enable Wi-Fi, enable flashlight) - available only in the USA", "web pages opened directly within the app". I heavily doubt it's taking these many years to roll out these changes; they probably simply don't care anymore, and never update these app store release notes.
There's always a chance of failure even for the most meticulous companies.
Now I'm not defending or excusing the company, but a singular event like this can happen to anyone and nothing is 100%.
If thorough investigation revealed poor quality control investment compared to what would be appropriate for a company like this, then we can say for sure.
Nobody ran this update
The update was pushed globally to all computers
With that alone we know they have failed the simplest of quality control methods for a piece of software as widespread as theirs. This is even excluding that there should have been some kind of error handling to allow the computer to boot if they did push bad code.
It's clear to me that CrowdStrike saw this as a data update vs. a code update, and that they had much more stringent QA procedures for code updates that they did data updates. It's very easy for organizations to lull themselves into this false sense of security when they make these kinds of delineations (sometimes even subconsciously at first), and then over time they lose site of the fact that a bad data update can be just as catastrophic as a bad code update. I've seen shades of this issue elsewhere many times.
So all that said, I think your point is valid. I know Crowdstrike had the posture that they wanted to get vulnerability files deployed globally as fast as possible upon a new threat detection in order to protect their clients, but it wouldn't have been that hard to build in some simple checks in their build process (first deploy to a test bed, then deploy globally) even if they felt a slower staged rollout would have left too many of their clients unprotected for too long.
Hindsight is always 20/20, but I think the most important lesson is that this code vs data dichotomy can be dangerous if the implications are not fully understood.
When you have the trifecta of regex, *argv packing and uninitialized memory you're reaching levels of incompetence which require being actively malicious and not just stupid.
It cannot have been a surprise to Crowdstrike that pushing bad data had the potential to bork the target computer. So if they had such an attitude that would indicate striking incompetence. So perhaps you are right.
Obviously the system wasn't very robust, as a simple, within specs change could break it. A company like CrowdStrike, which routinely deals with memory exploits and claims to do "zero trust" should know better.
As often, there is a good chance it is an organization problem. The team in charge of the parsing expected that the team in charge of the data did their tests and made sure the files weren't broken, while on the other side, they expected the parser to be robust and at worst, a quick rollback could fix the problem. This may indeed be the sign of a broken company culture, which would give some credit to the ex-employees.
From my limited understanding, the file was corrupted in some way. Lots of NULL bytes, something like that.
https://www.crowdstrike.com/wp-content/uploads/2024/08/Chann...
> Hindsight is always 20/20, but I think the most important lesson is that this code vs data dichotomy can be dangerous if the implications are not fully understood.
But it's not some new condition that the industry hasn't already been dealing with for many many decades (i.e. code vs config vs data vs any other type of change to system, etc.).
There are known strategies to reduce the risk.
They had previous bricked linux hosts earlier with a similar type of update.
So we also know that they don't learn from their mistakes.
The linux case is just _very_ different from the windows case. The mitigation steps that could have been taken to avoid the linux problem would not have helped for the windows outage anyways, the problems are just too different. The linux update was about an OS update breaking their program, while the windows issue was about a configuration change they made triggering crashes in their driver.
It's: a) an update, b) pushed out globally without proper testing, c) that bricked the OS.
It's an obvious failure mode that if you have a proper incident response process would be revealed from that specific incident and flagged for needing mitigation.
I do this specific thing for a living. You don't just address the exact failure that happened but try to identify classes of risk in your platform.
> Even if you could, you'd still need to deploy a fix before the OS update hits the customers, and anyone that didn't update would still be affected.
And yet the problem would still only affect Crowdstrike's paying customers. No matter how much you blame upstream your paying customers are only ever going to blame their vendor because the vendor had discretion to test and not release the update. As their customers should.
And again, you cannot prevent your vendor breaking you. Sure, you can magic some convoluted process to catch it asap. But that won’t help the poor sods who got caught in-between.
This is the point I would emphasize. A kernel module that parses configuration files must defend itself against a failed parse.
We don't really need that thorough of an investigation. They had no staged deploys when servicing millions of machines. That alone is enough to say they're not running the company correctly.
Guess which part took down much of the corporate world?
from Preliminary Post Incident Review at https://www.crowdstrike.com/falcon-content-update-remediatio... :
"CrowdStrike delivers security content configuration updates to our sensors in two ways: Sensor Content that is shipped with our sensor directly, and Rapid Response Content that is designed to respond to the changing threat landscape at operational speed.
...
The sensor release process begins with automated testing, both prior to and after merging into our code base. This includes unit testing, integration testing, performance testing and stress testing. This culminates in a staged sensor rollout process that starts with dogfooding internally at CrowdStrike, followed by early adopters. It is then made generally available to customers. Customers then have the option of selecting which parts of their fleet should install the latest sensor release (‘N’), or one version older (‘N-1’) or two versions older (‘N-2’) through Sensor Update Policies.
The event of Friday, July 19, 2024 was not triggered by Sensor Content, which is only delivered with the release of an updated Falcon sensor. Customers have complete control over the deployment of the sensor — which includes Sensor Content and Template Types.
...
Rapid Response Content is used to perform a variety of behavioral pattern-matching operations on the sensor using a highly optimized engine.
Newly released Template Types are stress tested across many aspects, such as resource utilization, system performance impact and event volume. For each Template Type, a specific Template Instance is used to stress test the Template Type by matching against any possible value of the associated data fields to identify adverse system interactions.
Template Instances are created and configured through the use of the Content Configuration System, which includes the Content Validator that performs validation checks on the content before it is published.
On July 19, 2024, two additional IPC Template Instances were deployed. Due to a bug in the Content Validator, one of the two Template Instances passed validation despite containing problematic content data.
Based on the testing performed before the initial deployment of the Template Type (on March 05, 2024), trust in the checks performed in the Content Validator, and previous successful IPC Template Instance deployments, these instances were deployed into production."
When there's 0day, how enterprisey you would like to catch the 0day?
This is a ludicrous position, and has been proven obviously false by the proceedings: all systems that were crashed by this critical failure were not, in fact, attacked with ransomware once the CS agent was un-installed (at great pain).
You don't want to be in a situation where you're taken hostage and asked hundred mills ransomeware just because you're too slow to mitigate the situation.
A patch should, at minimum:
1. Let the app run 2a. Block the offending behaviour 2b. Allow normal behaviour
Part 1. can be assumed if Parts 2a and 2b work correctly.
We know CrowdStrike didn't ensure 2a or 2b since the app caused the machine to reboot when the patch caused a fault in the app.
CrowdStrike's Root Cause Analysis, https://www.crowdstrike.com/wp-content/uploads/2024/08/Chann..., lists what they're going to do:
====
Mitigation: Validate the number of input fields in the Template Type at sensor compile time
Mitigation: Add runtime input array bounds checks to the Content Interpreter for Rapid Response Content in Channel File 291 - An additional check that the size of the input array matches the number of inputs expected by the Rapid Response Content was added at the same time. - We have completed fuzz testing of the Channel 291 Template Type and are expanding it to additional Rapid Response Content handlers in the sensor.
Mitigation: Correct the number of inputs provided by the IPC Template Type
Mitigation: Increase test coverage during Template Type development
Mitigation: Create additional checks in the Content Validator
Mitigation: Prevent the creation of problematic Channel 291 files
Mitigation: Update Content Configuration System test procedures
Mitigation: The Content Configuration System has been updated with additional deployment layers and acceptance checks
Mitigation: Provide customer control over the deployment of Rapid Response Content updates
====
I’d consider staggering a rollout to be the absolute basics of due diligence.
Especially when you’re building a critical part of millions of customer machines.
Specifically because this is about fighting against malicious actors, where time can be of essence to deploy some sort of protection against a novel threat.
If there's deadlines that you can go over, and nothing bad happens, for sure. Always have canary releases, and perfect QA, monitoring everything thoroughly, but I'm just saying, there can be cases where damage that could be done if you don't act fast enough, is just so much worse.
And I don't know that it wasn't the case for them. I just don't know.
This is severely overstating the problem: an extra few minutes is not going to be the difference between their customers being compromised. Most of the devices they run on are never compromised, because anyone remotely serious has defense in depth.
If it was true, or even close to true, that would make the criticism more rather than less strong. If time is of the essence, you invest in things like reviewing test coverage (their most glaring lapse), fuzz testing, and common reliability engineering techniques like having the system roll back to the last known good configuration after it’s failed to load. We think of progressive rollouts as common now but they got to get that mainstream in large part because the Google Chrome team realized rapid updates are important but then asked what they needed to do to make them safe. CrowdStrike’s report suggests that they wanted rapid but weren’t willing to invest in the implementation because that isn’t a customer-visible feature – until it very painfully became one.
Before the incident, if you asked a customer if they would like to get updates faster even if it means that there is a remote chance of a problem with them... I bet they'd still want to get updates faster.
/s
close to nothing /s
If you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html and taking the intended spirit of the site more to heart, we'd be grateful.
I thought the new code was actually installed, the running part depends on the script input...?
However, every company yes-man is paid to be a yes-man and will speak in favor of the company without exception - that literally is the job. Otherwise they will be fired and will join the ranks of the aforementioned people.
So logically it makes more sense for me to believe the former more than the latter. The two-sides are not equivalent (as you may have alluded) in term of trustworthiness.
Mostly disappointed.
That makes it easier to trust one side over another.
If the employees fucked up then you’ll say the company still fucked up because it wasn’t managing the employees well.
And then in that situation you’ll still believe the lying employees who say its the company’s fault while leaving out their culpability.
Yes, I'm more likely to leave reviews if I'm unsatisfied. Yes, people are more likely to leave CS if they were unhappy. Biased data, but still useful data.
I bet my ass UX is not anywhere close to the low-level OS team.
UX is definitely embedded in the App level team but not in low-level.
Granted, a company that isn’t focused on the user experience as much as it is on other things might not prioritise this as much in the first place.
Software security and quality is the responsibility of everyone on the team. A good UX designer should be thinking of ways a user can escape the typical flow or operate in unintended ways and express that to testers. And in decisions where management is forcing untested patches everyone should chime in.
Except the fact that CrowdStrike fucked up the one thing they weren't supposed to fuck up.
So yeah, at this point I'm taking the ex-employees' word, because it confirms the results that we already know -- there is no way that update could have gone out had there been proper "safety first" protocols in place and CrowdStrike was "meticulous".
To you and me, maybe. To the insurers and airlines paying out over the problem, maybe not.
Source: me, a developer who also codes in free time and notices how bad fs perf is especially.
I've had the CrowdStrike sensor, and my current company is using cyberhaven.
So.. while 2 data points don't technically make a pattern, it does begin to raise suspicion.
No, what we have is a publication who is claiming that the people they talked to were credible and had points that were interesting and tended to match one another and/or other evidence.
You can make the claim that Semafor is bad at their jobs, or even that they're malicious. But that's a hard claim to make given that in the paragraph you've quoted they are giving you the contrary evidence that they found.
And this is a process many of us have done informally. When we talk to one ex-employee of a company, well maybe it was just that guy, or just where he was in the company. But when a bunch of people have the same complaint, it's worth taking it much more seriously.
I would expect any contractor that may have worked for CrowdStrike, or done something like a third-party audit, would be under an NDA covering their work.
Who's left to speak out with any meaningful details?
Is anyone really surprised or learned any new information? For us that have worked for tech companies, this is one of those repeating complaints that you hear across orgs that indicates a less than stellar engineering culture.
I've worked with numerous F500 orgs and I would say 3/5 orgs that I worked in, their code was so bad that it made me wonder how they haven't had a major incident yet.
Employees don't typically have much reputation to ruin. I am perfectly content putting this on their shoulders.
Ex-employees said bugs caused the log monitor to drop entries. Crowdstrike responded the project was never designed to alert in real time. But Crowdstrike's website currently advertises it as working in real time.
Ex-employees said people trained to monitor laptops were assigned to monitor AWS accounts with no extra training. Crowdstrike replied that "there were no experienced ‘cloud threat hunters’ to be had" in 2022 and that optional training was available to the employees.
This type of article - built upon disgruntled former employees - is worth about as much as the apology GrubHub gift card.
Look, I think just as poorly about CrowdStrike as anyone else out there... but you can find someone to say anything, especially when they have an axe to grind and a chance at some spotlight. Not to mention this guy was a designer and wouldn't be involved in QC anyway.
> Of the 24 former employees who spoke to Semafor, 10 said they were laid off or fired and 14 said they left on their own. One was at the company as recently as this summer. Three former employees disagreed with the accounts of the others. Joey Victorino, who spent a year at the company before leaving in 2023, said CrowdStrike was “meticulous about everything it was doing.”
So basically we have nothing.