Honest question, what should they do? Uninstall the company and give up? Ignoring their actual response, what is a good response to this?

If my company causes billions in damages and endangers human lives, I can't imagine why my company isn't bankrupted and dissolved.

Ok, but the market makes that decision, not the company. Crowdstrike has no choice but to accept the sentence the market hands it. It’s just that the market appears to have sentenced it to…barely anything. Blame those still using CrowdStrike after this incident.

> Blame those still using CrowdStrike after this incident.

I think you'd have to ask "why are they still required to use CrowdStrike or any AV provider?" I think once you find the answers to these questions you realize this is not a properly functioning product market.

How you can then build a publicly traded company on the back of a complete and total lie is another subject, but it's certainly also implicated in the above questions.

If your company causes damage at society-scale (hell, even if it does major damage to one person's life), the state should be ready to intervene and make the company pay the tab for the damage they caused? Like, that doesn't sound very controversial.

Yea. Their contracts likely have clauses for all of that. I say likely, but we already know this is true because it's come out.

The thing is, crowdstrike isn't the only incompetent party here. Many major companies (looking at Delta) probably made it worse for themselves with a very poor response after.

So should crowdstrike pay beyond a reasonable measure because of Delta's poor response?

No contract clause can protect you from a gross negligence tort.

(Or equivalent in one's respective civil law system.)

This might be the easiest gross negligence tort case to show and litigate-- still hard but if everyone starts the lawsuits they can not pull the contract to protect them. They will try of course and they will fail in most but the obvious cases.

What you can not sue them for is not forseeable damages -- e.g. I lost my dream job because the computer died during the interview. But ceasing operations of a company is generally fair game. And plaintiffs can argue that no reasonable person could forsee and mitigate against this disaster so the failure is not due to plaintiff's "fault" negligence.

Reckless typically requires conscious disregard of risk. Arguably, that would require Crowdstrike emails from programmers saying “this is risky, we need to test it” and management responding “F it! We’ll do it live!”

If nobody in CS realized how dangerous their process was, it’s not reckless.

That's interesting but my sniff test isn't passing. "Reckless driving" doesn't require me to know it's a bad idea to do 100 miles per hour in a 25, it is reckless whether I realize it or not right? IANAL but the only thing I can think of requiring knowing to be at fault is slander, at least in the USA.

Actually generally the legal system would decide that, not “the market”.

I.e. investors have assigned roughly zero probability to CrowdStrike bearing the full cost of this incident, and set the market price accordingly.

The market as a tool for punishing bad players is far from perfect. It's why we still have monopolies and see consumer antitrusts and other similar legal suits in court. Advocating for shifting blame to customers still using Crowdstrike is ignorant of the problem and further signals a dishonest approach to the issue at hand.

Equifax, for example. They probably caused way more damage leaking all our info

The company should be fined so heavily that they become a goverment asset.

With current shareholders wiped out entirely, it should then be re-listed.

IMO the company that this should have happened to is PG&E. I think California could have forced them into liquidation and bought their assets. No bailout required, complete loss for shareholders, and CA could potentially have fixed much of its disastrous utility situation at a reasonable price.

A company-ending fine or judgment against Crowdstrike wouldn’t come with any great reason for a public takeover — Crowdstrike could cease to exist and the overall ecosystem would be fine.

Outside of Alaska and Wyoming, Silicon Valley has the worst power and internet of anywhere I've ever lived (worse than AR, MN, and ND by a longshot), measured either in incremental cost or uptime/availability. The fact that PGE keeps requesting additional rate increases "for fire safety" and immediately kicks those back to shareholders isn't a great look either.

Hide themselves from the face of the world for at least 5 years until people somewhat forget about them.

Because people on the receiving end are the same - they accepted and rolled out the update without even as much as “canarying” it. SolarWind was the same - the customers weren’t bothered even by mismatched integrity hashes. It is a tacit pact in our industry - we all screw up and cut the slack to each other. Who will cast the first stone?

> they accepted and rolled out the update without even as much as “canarying” it.

Well, no; AIUI part of the problem was precisely that this update was pushed in such a way that it skipped any canary system in place. There might be a separate conversation to question what percentage of their users were taking advantage of its staged rollout features, but it's rather immaterial when the incident in question bypassed them even if users had configured it sensibly.

But the customer installed CS software could do this. So they are partly to blame. I do not think you will find that tesla would allow a third party update to its car. Or a oil rig would allow third party updates to critical parts of its systems. So its understanding the context. I think a lot of places this is an risk that is ok. But maybe not everywhere. And I hope some companies with critical systems will learn from this

> But the customer installed CS software could do this. So they are partly to blame.

It depends on if/how it was communicated. If there's a big red box in the user manual that says, "this software might take updates that completely bypass any phased rollout you configure", then yes it was probably irresponsible to use it. If, however, the software lets you configure phased rollouts and fails to mention that they might just get ignored, then I don't see how the customer can be blamed at all. (And in both cases, if CS shipped such an update with exactly zero testing whatsoever, which strains credulity but is what I've read, then they still get most of the blame.)

Crowdstrike can force-push an update at any time of their choosing that the connected device will grab and load, is my understanding.

Don’t you see that you’re only enforcing my point?

No, because "canary" in the context that you used it, has a specific meaning. If you believe they should have tested CrowdStrike more or been more skeptical of their claims before licensing, that's independent of the user/administrators doing canary-style testing.

What’s the default? And what did their technical account manager recommend? My guess, no canary ring.

Crowdstrike should be held accountable but so should any reasonably sized enterprise that allows code to be pushed to their whole enterprise without testing. All the large enterprises I've worked for required Windows patches to be tested before being pushed to production why are crowdstrike updates treated differently?

It clearly states in the post mortem that rapid response updates were pushed globally without any say from the customer: https://www.crowdstrike.com/wp-content/uploads/2024/08/Chann...

So in theory if a company like crowstrike were compromised by criminals/disgruntled employee/NSA the impact could not be avoided?

Given the cybersecurity landscape it is not unusual for security software updates to be pushed globally without the option to test or even stagger them. When a potent new vulnerability becomes public knowledge (or semi-insider knowledge) and especially if there's already a PoC available, organizations only have minutes to a few hours before threat actors begin utilizing it.

APTs and organized crime groups have 24/7 staff to weaponize and integrate new vulnerabilities into their workflow as rapidly as possible, or have other contracted other groups to provide this service.