Re:Technical Details on Today's Outage
by dalyons on Saturday July 20, 2024 @10:54AM
(#41016900)
A couple of hours is a long time in the world of automated attacks
[ Parent
| on HN ]
Re:Technical Details on Today's Outage
by mynameisvlad on Saturday July 20, 2024 @11:28AM
(#41017142)
A couple of hours is absolutely nothing compared to the massive worldwide effort that many people have to put in to fix the problem of a company’s shitty product and release practices.
This is inexcusable, point blank. “A couple of hours is a long time” is not a valid excuse when the alternative, as clearly evidenced, is millions of computers and critical systems simultaneously failing hard.
This might have been different if it was a small subset of computers, but this clearly could have been caught in minutes with any sort of sensible testing or canary rollout practices.
Re:Technical Details on Today's Outage
by notepad0x90 on Saturday July 20, 2024 @09:35PM
(#41021694)
I'm guessing they didn't expect content updates to cause such an impact, they've been doing this for 15 years, it is that uncommon. a couple of hours in their world is a long time because their concern is protecting customers as soon as possible. I'm sure they'll do all kinds of tests going forward and be transparent about it. Keep in mind how easy it is for you or I to come to conclusions without understanding or knowing the context they operate in, maybe it will be more clear soon enough.
Re:Technical Details on Today's Outage
by RaftPeople on Saturday July 20, 2024 @11:44AM
(#41017265)
It only takes a couple of minutes if you first update your on-site set of LIVE systems sitting there to detect a problem.
If problem encountered, don't send it out to everyone else.
Re:Technical Details on Today's Outage
by TheOtherHobbes on Saturday July 20, 2024 @11:09AM
(#41017010)
Not as long as the weeks it's going to take to undo this.
Re:Technical Details on Today's Outage
by mardifoufs on Saturday July 20, 2024 @01:43PM
(#41018201)
Then they should make their testing pipelines even faster, and make sure that they can go from detecting a new threat->tested definition file as quickly as possible. You genuinely cannot skimp on testing in this case. It's inherent to the update, threat protection and not breaking their consumers systems should be non-negotiable for a release. That means testing before deploying. If they can't do it fast enough, their product is broken.
An automated attack would struggle to reach the level of destruction that this failure had due the scale of Crowdstrike deployment and the direct update vector and kernel mode failure. Even with the most critical type of remote vulnerability it would be difficult to achieve anything approaching this level of damage, and for all we know (and by all probabilities) this update was addressing a much less severe vulnerability.