News for hackers. Stuff that matters.
Re:CrowdStrike accepting the PwnieAwards for "most epic fail... by jedberg on Sunday August 11, 2024 @01:14PM (#41217689)
Delta threatened to sue them for their $500M loss. Crowdstrike replied (publicaly) pointing out that their contract limits Crowdstrike's liability to single digit millions.

Then then gave them a list of things they would seek in discovery, such as their backup plans, failover plans, testing schedules and results, when their last backup recover exercise was, etc.

Basically, they said, "if you sue us, we will dig so deep into your IT practices that it will be more embarrassing for you than us and show that you were at fault".

Re:CrowdStrike accepting the PwnieAwards for "most epic fail... by mikeocool on Sunday August 11, 2024 @02:57PM (#41218518)
It really seems funny that Crowdstrike’s defense is basically “you should have been better prepared for us to knock all of your systems offline.”

It’s probably true, but seems like an odd stance to take from a PR perspective or a “selling other clients in the future” perspective.

Re:CrowdStrike accepting the PwnieAwards for "most epic fail... by jedberg on Sunday August 11, 2024 @02:59PM (#41218538)
In the case of Delta, their outage was much longer than everyone else because they refused help from both Crowdstrike and Microsoft. So their defense is basically "the damages could have been mitigated if you'd listened to us".
Re:CrowdStrike accepting the PwnieAwards for "most epic fail... by NavinF on Sunday August 11, 2024 @03:15PM (#41218623)
> they refused help from both Crowdstrike and Microsoft

Link?

Anyway I find it highly amusing that Delta is seeking damages from Microsoft even though Microsoft had nothing to do with it.

Re:CrowdStrike accepting the PwnieAwards for "most epic fail... by jedberg on Sunday August 11, 2024 @03:23PM (#41218668)
There are many articles about them refusing help, but here is one:

https://www.theverge.com/2024/8/6/24214371/microsoft-delta-l...

Re:CrowdStrike accepting the PwnieAwards for "most epic fail... by cratermoon on Sunday August 11, 2024 @06:59PM (#41219869)
Delta's position is the Microsoft actively recommended and coordinated with CrowdStrike to the extent that they are co-responsible for outcomes. In a large enterprise like Delta, the vendors do work together in deployment and support. Yes, there's often a great deal of finger-pointing between vendors when something like this happens, but in general vendors so intimately linked have each other on speed-dial. It would not shock me to learn that Delta has email or chat threads involving CrowdStrike, Microsoft, and Delta employees working together during rollouts and upgrades, prior to this event.

As far as refusing help, why is that funny? If someone does something stupid and knocks you down, it's perfectly reasonable to distrust the help they offer, especially if that help requires giving them even more trust than what they've already burned.

Re:CrowdStrike accepting the PwnieAwards for "most epic fail... by hamstercat on Sunday August 11, 2024 @08:06PM (#41220137)
Changing vendors and choosing one that's more reliable is a perfectly sensible outcome of this situation once your system are back up and you're no longer hemorrhaging money.

During an ongoing incident, when all of your operations are down, is not the time for it though. If you think there's even a 1% chance that the help can help, you should probably take it and fix your immediate problem. You can re-evaluate your decisions and vendor choices after that.

Re:CrowdStrike accepting the PwnieAwards for "most epic fail... by troyvit on Monday August 12, 2024 @11:34AM (#41225625)
> If someone does something stupid and knocks you down, it's perfectly reasonable to distrust the help they offer, especially if that help requires giving them even more trust than what they've already burned.

Yeah it smacks of Experian offering you a year of "free identity theft protection" after having lost your personal data in a breach.

Re:CrowdStrike accepting the PwnieAwards for "most epic fail... by cnlwsu on Sunday August 11, 2024 @03:32PM (#41218732)
That’s kinda what aws tells people when its services go down. If your backend can’t take a short outage without weeks of recovery then it’s just a matter of time.
Re:CrowdStrike accepting the PwnieAwards for "most epic fail... by cratermoon on Sunday August 11, 2024 @06:53PM (#41219850)
That's kind of typical of how much companies have been allowed to externalize costs. It's never about how the company at fault should have done better, rather it typically boils down to some variant of "the free markets provided you with a choice about who you trust and it was up to you to collect and evaluate all the information available to make your choices".
Re:CrowdStrike accepting the PwnieAwards for "most epic fail... by JumpCrisscross on Sunday August 11, 2024 @01:19PM (#41217724)
> they said, "if you sue us, we will dig so deep into your IT practices that it will be more embarrassing for you than us and show that you were at fault"

But CrowdStrike said this publicly. If they’d privately relayed it to Delta, it would have been genuine. By performatively relaying it, however, it seems they’re pre-managing optics around the expected suit.

Re:CrowdStrike accepting the PwnieAwards for "most epic fail... by wjnc on Sunday August 11, 2024 @02:42PM (#41218401)
It’s an argument that hits home at any bigcorp where the execs are entertaining the thought of suing CrowdStrike. Making it public once is a lot more effective than relaying it privately a hundred times. I expect most liability to come from abroad, where parts of the contract might be annulled because not in line with local law. But still I don’t expect it. CrowdStrike delivered the service they promised. The rest is on the customers IT. Hand over the keys and your car may be driven.
Re:CrowdStrike accepting the PwnieAwards for "most epic fail... by JumpCrisscross on Sunday August 11, 2024 @02:45PM (#41218422)
> It’s an argument that hits home at any bigcorp where the execs are entertaining the thought of suing CrowdStrike

Maybe? Discovery is a core element of any lawsuit. It’s also a protected process: you can’t troll through confidential stuff with an intent to make it public to damage the litigant.

If anything, I could see Delta pointing to this statement to restrict what CrowdStrike accesses and how [1]. (As well as with the judge when debating what gets redacted or sealed.)

[1] https://www.fjc.gov/sites/default/files/2012/ConfidentialDis...

Re:CrowdStrike accepting the PwnieAwards for "most epic fail... by wjnc on Monday August 12, 2024 @12:40AM (#41221210)
Thank you. Nice read. Even given a protective order to keep discovery confidential, the ensuing discussion about the clients lacking IT-policies that exacerbated this crisis is public.

Most entertaining would be the discussion where CrowdStrike would argue that based on common IT-risk criteria, you should never hand over the keys to an unaudited party not practicing common IT-risk best practices and (thus) the liability is on the organization. Talk about CrowdStrike managing risks worldwide. They are doing it right now!

Re:CrowdStrike accepting the PwnieAwards for "most epic fail... by gnfargbl on Sunday August 11, 2024 @01:21PM (#41217740)
Or attempting to discourage it from becoming a pile-on.
Re:CrowdStrike accepting the PwnieAwards for "most epic fail... by magic_man on Sunday August 11, 2024 @01:34PM (#41217836)
It doesn't matter it was 100% crowdstrikes fault. Surprised its still worth 60billion dollars.
Re:CrowdStrike accepting the PwnieAwards for "most epic fail... by from-nibly on Sunday August 11, 2024 @01:42PM (#41217904)
Part of the problem is assuming you can pay a contract to shift your liability completely away.
Re:CrowdStrike accepting the PwnieAwards for "most epic fail... by unyttigfjelltol on Sunday August 11, 2024 @02:12PM (#41218188)
Right, the risk structure presumably protects the vendor if just one customer sues, even if the amount of damages claimed is astronomical. Because vendors try to disclaim bet-the-company liability on a single contract.[1] The vendor's game is to make sure the rest of the customer base does not follow this example, because as noted in the linked article while vendors don't accept bet-the-company liability on each contract (or try not to), they do normally have some significant exposure measured in multiples of annual spend.

[1] https://www.gs2law.com/blog/current-trends-in-liability-limi...

Re:CrowdStrike accepting the PwnieAwards for "most epic fail... by iwontberude on Sunday August 11, 2024 @02:02PM (#41218104)
Well if MSFT knew how to write MSAs Crowdstrike would have become property of Microsoft.
Re:CrowdStrike accepting the PwnieAwards for "most epic fail... by TeMPOraL on Sunday August 11, 2024 @06:58PM (#41219865)
The assumption is not only perfectly valid, it's the very reason such contracts are signed in the first place! It's what companies want to buy, and it's what IT security companies exist to sell.
Re:CrowdStrike accepting the PwnieAwards for "most epic fail... by from-nibly on Monday August 12, 2024 @11:33AM (#41225613)
Yes, I know that's what everyone wants/thinks, but you actually can't do it. Because at the end of the day, you chose the vendor. So you are still liable for all of it.
Re:CrowdStrike accepting the PwnieAwards for "most epic fail... by pknomad on Sunday August 11, 2024 @02:07PM (#41218139)
Yes and no.

Crowdstrike was the executioner of this epic fail for sure but their archaic infra practices made it even worse. Both Crowdstrike and Microsoft CEOs reached out only to be rebuffed by Delta's own. If I was the CEO - I'd accept any help I can get while you have the benefit of the public opinion.

/tin-foil-hat-on Flat out refusal for help makes me think there are other skeletons in the closet that makes Delta look even worse /tin-foil-hat-off

Re:CrowdStrike accepting the PwnieAwards for "most epic fail... by JumpCrisscross on Sunday August 11, 2024 @02:51PM (#41218472)
> I was the CEO - I'd accept any help I can get while you have the benefit of the public opinion

I’d reserve judgement. Delta may have been cautious about giving the arsonists a wider remit.

Re:CrowdStrike accepting the PwnieAwards for "most epic fail... by pknomad on Sunday August 11, 2024 @06:39PM (#41219796)
Using your analogy - if MS/CS are the arsonists, then Delta are the landlords unsafely storing ammonium nitrate in their own warehouse.

Their lack of response to MS/CS isn't coming from a place of reducing potential additional problems but trying to shield their own inadequacies while a potential lawsuit is brewing in the background.

https://www.reuters.com/technology/microsoft-blames-delta-it...

Re:CrowdStrike accepting the PwnieAwards for "most epic fail... by TeMPOraL on Sunday August 11, 2024 @07:05PM (#41219894)
In this case, the fire was an accident, and the arsonists happen to be the expert firefighters, and they're very motivated to fix their mistake. They're still the experts in all stuff fire, whereas Delta is not.
Re:CrowdStrike accepting the PwnieAwards for "most epic fail... by ImJamal on Sunday August 11, 2024 @11:13PM (#41220888)
It doesn't seem like arsonist is the right word. It implies it was intentional, which as far as I can tell there is no proof of.

I think the more accurate description would be some firefighters were doing a controlled burn. The burn got out of controlled and then you say that you don't want the firefighters help in put out the fire.

Re:CrowdStrike accepting the PwnieAwards for "most epic fail... by evilduck on Sunday August 11, 2024 @03:49PM (#41218865)
If you held the view that CrowdStrike and Microsoft were inherently to blame for the problem why would you trust them to meaningfully help? At best they're only capable of getting you right back into the same position that left you vulnerable to begin with.
Re:CrowdStrike accepting the PwnieAwards for "most epic fail... by pknomad on Sunday August 11, 2024 @06:36PM (#41219784)
Same reason why an aircraft manufacturing company would get involved in a NTSB investigation when there is an airplane crash. Just because they messed up one or more things (i.e. MCAS on MAX) doesn't mean they can't provide expertise or additional resources to at least help with the problem.

Your take also casually disregards the fact that Delta took an extraordinary time to recover from the problem when the other companies recovered (albeit slowly). This is the point that I'm getting at. It isn't that CS and MS aren't culpable for the outage; it's that DAL also contributed to the problem by not adequately investing in its infra.

Re:CrowdStrike accepting the PwnieAwards for "most epic fail... by cratermoon on Sunday August 11, 2024 @07:08PM (#41219907)
> Same reason why an aircraft manufacturing company would get involved in a NTSB investigation when there is an airplane crash

Key difference here is that the NTSB is third party with force of law behind it. The victims in the crash – airlines and passengers – aren't rushing to the aircraft manufacturer to come fix things. Quite the opposite: the NTSB and FAA have the authority to quarantine a crash site and ensure nobody tampers with the evidence. Possible tampering with black boxes was an issue in the investigation of Air France Flight 296Q.

Re:CrowdStrike accepting the PwnieAwards for "most epic fail... by hamstercat on Sunday August 11, 2024 @08:18PM (#41220177)
Being to blame is different than being actively trying to sabotage you. Many companies will be re-evaluating their relationship after this problem happened, but doing that while your systems aren't functional seems counter-productive.
Re:CrowdStrike accepting the PwnieAwards for "most epic fail... by cyanydeez on Sunday August 11, 2024 @02:17PM (#41218221)
Weirdly, we live in a society
Re:CrowdStrike accepting the PwnieAwards for "most epic fail... by tedunangst on Sunday August 11, 2024 @04:51PM (#41219236)
Seems fair. Delta didn't privately relay their intentions.
Re:CrowdStrike accepting the PwnieAwards for "most epic fail... by richardw on Sunday August 11, 2024 @02:18PM (#41218231)
I’m not sure anything else was material given that the machines were bricked and client roll-out approaches were evaded by Crowdstrike. What client actions would have helped?

Surely someone is looking at a class action? People died. The contract can’t make that everyone else’s problem, can it?

Re:CrowdStrike accepting the PwnieAwards for "most epic fail... by gizmo686 on Sunday August 11, 2024 @03:19PM (#41218652)
Sure it can. If every rock climbing company in the country decides that climbing ropes are too expensive and instead decide to by rope from the local hardware store, and that rope has a warning reading "not for use when life or valuable property is at risk", then it is 100% on those climbing companies when people die, because they were using a product in a situation that it was simply not suitable for.

The details, of course, depend on the contract and claims that Crowdstrike made. But, in the abstract, you are not responsible for making your product suitable for any use that anyone decides to use it for.

If a hospital wants to install software on their life critical infastructure, they are supposed to buy software that is suitable for life critical infastructure.

Re:CrowdStrike accepting the PwnieAwards for "most epic fail... by lupire on Sunday August 11, 2024 @03:18PM (#41218643)
If someone's life depends on a networked Windows (or any similar OS) machine you chose to run for that purpose, you are the criminal.
Re:CrowdStrike accepting the PwnieAwards for "most epic fail... by 1over137 on Sunday August 11, 2024 @05:50PM (#41219546)
Indeed. But this is how hospitals run.
Re:CrowdStrike accepting the PwnieAwards for "most epic fail... by trentnix on Sunday August 11, 2024 @02:05PM (#41218129)
They should be providing all that information regularly to auditors anyway. If they don’t have it handy, then their IT leadership should be replaced.
Re:CrowdStrike accepting the PwnieAwards for "most epic fail... by reaperducer on Sunday August 11, 2024 @01:28PM (#41217795)
Delta threatened to sue them for their $500M loss. Crowdstrike replied (publicaly) pointing out that their contract limits Crowdstrike's liability to single digit millions.

Delta's move seems like an attempt to assuage shareholders and help the C.E.O. save face.

Crowdstrike shouldn't be afraid of Delta. Crowdstrike should be afraid of the insurance companies that have to pay out to those businesses that have coverage that includes events like this.

Even if the payout to a company is $10,000, a big insurance company may have hundreds or thousands of similar payouts to make. The insurance companies won't just let that go; and they know exactly what to look for, how to find it, and have the people, lawyers, and time to make it happen.

Crowdstrike will get its day of reckoning. It won't be today. And it probably won't be public. But the insurance companies will make sure it comes, and it's going to hurt.

Re:CrowdStrike accepting the PwnieAwards for "most epic fail... by JumpCrisscross on Sunday August 11, 2024 @01:32PM (#41217817)
> the insurance companies will make sure it comes, and it's going to hurt

It could be as simple as a reinsurer refusing to renew coverage if a company uses CrowdStrike.

Re:CrowdStrike accepting the PwnieAwards for "most epic fail... by vladvasiliu on Sunday August 11, 2024 @01:36PM (#41217849)
Which would be funny, since many companies are putting up with Crowdstrike to make insurers happy.
Re:CrowdStrike accepting the PwnieAwards for "most epic fail... by dredmorbius on Sunday August 11, 2024 @02:36PM (#41218363)
Availability (or not) of insurance coverage is surprisingly effective in enabling or disabling various commercial ventures.

The penny dropped for me whilst reading James Burke's Connections on the exceedingly-delayed introduction of the lateen-rigged sail to Europe, largely on the basis that the syndicates which underwrote (and insured) shipping voyages wouldn't provide financing and coverage to ships so rigged.

Far more recently we have notions of redlining for both mortgage lending and insurance coverage (title, mortgage, property, casualty) in inner-city housing and retail markets. Co-inventor of packet-based switching writes of his parents' experience with this in Philadelphia:

"On the Future Computer Era: Modification of the American Character and the Role of the Engineer, or, A Little Caution in the Haste to Number" (1968)

<https://www.rand.org/pubs/papers/P3780.html> (footnote, p. 6).

Similarly, government insurance or guarantees (Medicare, SSI, flood insurance, nuclear power plants) has made high-risk prospects possible, or enabled effective services and markets, where laissez-faire approaches would break down.

I propose that similar approaches to issues such as privacy violation might be worth investigating. E.g., voiding any insurance policy over damages caused through the harmful use or unintended disclosure of private information. Much of the current surveillance-capitalism sector would instantly become toxic. The principle current barriers to this are that states themselves benefit through such surveillance, and of course the current industry is highly effective at lobbying for its continuance.

Re:CrowdStrike accepting the PwnieAwards for "most epic fail... by selimthegrim on Sunday August 11, 2024 @07:24PM (#41219963)
That’s interesting because on the TV episode, it states that insurers wanted the risk of piracy spread out over many smaller ships that would be lateen rigged. I have one of the Connections books, so I’ll check to see if this is covered in it https://youtu.be/1NqRbBvujHY?si=WfysDHPLhSJkGhzd
Re:CrowdStrike accepting the PwnieAwards for "most epic fail... by dredmorbius on Sunday August 11, 2024 @09:59PM (#41220607)
Interesting discrepancy, yes. I'm pretty sure of my recollection of the book.

It may be that the opportunity to diversify risk (over more smaller ships) overcame the reluctance to adopt new, untested and/or foreign technology.

Re:CrowdStrike accepting the PwnieAwards for "most epic fail... by selimthegrim on Sunday August 11, 2024 @11:42PM (#41220998)
It doesn’t explicitly say insurers but it’s a pretty small logical leap from the wording (the timeframe is also c. 11th-12th century so could be before formal insurers)
Re:CrowdStrike accepting the PwnieAwards for "most epic fail... by dredmorbius on Sunday August 11, 2024 @11:51PM (#41221040)
Right.

The books and video scripts also differ amongst Burke's various series. I'll see if I can find a copy of the text to compare.

Re:CrowdStrike accepting the PwnieAwards for "most epic fail... by VirusNewbie on Sunday August 11, 2024 @02:14PM (#41218204)
They might find out delta does embarrassing things like not testing out of bounds array access or does global deployments without canarying.
Re:CrowdStrike accepting the PwnieAwards for "most epic fail... by mc32 on Sunday August 11, 2024 @02:08PM (#41218146)
That’s odd. One is an internal process which has no obligation to an external party, and the other one who is specifically responsible for being liable for any repercussions due to deviating from their own SDLC process[1]they totally skipped themselves?

If I were Delta, I’d get other affected parties and together sue CrowdStrike and get all their dirty laundry out in the open.

[1] I haven’t checked but they used to list all their ISO certs, etc. Wonder if those get revoked for such glaring violations…

Re:CrowdStrike accepting the PwnieAwards for "most epic fail... by reaperman on Sunday August 11, 2024 @02:14PM (#41218209)
Civil suits focus in a large way on determining how much damage is each party’s fault. So Crowdstrike would be saying “Of this $500M in damages, x% was from your own shitty practices not from our mistake”. Thats why it’s all pertinent.
Re:CrowdStrike accepting the PwnieAwards for "most epic fail... by otterley on Monday August 12, 2024 @02:08AM (#41221519)
Correct. The legal term is “contributory negligence.”
Re:CrowdStrike accepting the PwnieAwards for "most epic fail... by MattGaiser on Sunday August 11, 2024 @02:22PM (#41218267)
> One is an internal process which has no obligation to an external party

Delta has obligations to their passengers and similarly sidesteps screw ups with similar contractual provisions. How much would Delta owe for not following similar IT practices? Do they now owe customers for their IT failings? Should customers now get to sue Delta for damages related to their poor IT recovery compared to other airlines?

Re:CrowdStrike accepting the PwnieAwards for "most epic fail... by mc32 on Sunday August 11, 2024 @02:24PM (#41218292)
Sure but that’d be something passengers could bring up in a suit against Delta, not someone like CS, who themselves obviously skipped their own internal SDLC and whatever other ISO certs they prominently advertised on their website.
Re:CrowdStrike accepting the PwnieAwards for "most epic fail... by jc2jc on Sunday August 11, 2024 @02:46PM (#41218425)
I assume the argument is that if they can show negligence in their IT practices, then the $500 million in damages can't be all attributed to CrowdStrike's failure.
Re:CrowdStrike accepting the PwnieAwards for "most epic fail... by dredmorbius on Sunday August 11, 2024 @02:37PM (#41218369)
Crowdstrike's discovery process would greatly aid in passenger or general-public suits against Delta.
Re:CrowdStrike accepting the PwnieAwards for "most epic fail... by icehawk on Sunday August 11, 2024 @05:52PM (#41219558)
I'd LOVE to see Crowdstrike do this. The last time I dealt with the specifics of this sort of validation testing for security software was a decade and from what I saw in the RCA Delta can just keep pointing out that whatever they had worked until Crowdstrike failed to understand that the number 20 and the number 21 are not the same:

The new IPC Template Type defined 21 input parameter fields, but the integration code that invoked the Content Interpreter with Channel File 291’s Template Instances supplied only 20 input values to match against. This parameter count mismatch evaded multiple layers of build validation and testing, as it was not discovered during the sensor release testing process, the Template Type (using a test Template Instance) stress testing or the first several successful deployments of IPC Template Instances in the field.

This combined with the lack of partitioning updates, makes me draw the conclusions they're missing table stakes WRT to validation.

Re:CrowdStrike accepting the PwnieAwards for "most epic fail... by mikhael28 on Sunday August 11, 2024 @09:30PM (#41220469)
Wtf how do you not check for ‘quantity of arguments’ in QA testing?
Re:CrowdStrike accepting the PwnieAwards for "most epic fail... by otterley on Monday August 12, 2024 @02:06AM (#41221506)
That’s not the way legal process works. CrowdStrike might be permitted to conduct discovery, but that won’t entitle them to share what they might find with the public, embarrassing or otherwise. Business records and other sensitive information relating to parties in civil matters are frequently sealed.