News for hackers. Stuff that matters.
Re:Technical Details on Today's Outage by chris_nielsen on Friday July 19, 2024 @10:38PM (#41013590)
Yeah “how this logic flaw occurred” is the wrong question.

How a common bug was rolled out globally with no controls, testing, or rollback strategy is the right question

Re:Technical Details on Today's Outage by YZF on Friday July 19, 2024 @11:09PM (#41013725)
They're all good questions. The thing that reads the config should have been fuzz tested with something like AFL. Likely should have a lot more tests. Maybe shouldn't run in a device driver. There's almost no doubt there are engineering process and culture issues here.

And then absolutely the release process.

Rollback is hard I guess once your OS can't boot.

Re:Technical Details on Today's Outage by cube00 on Saturday July 20, 2024 @12:10PM (#41017488)
> Rollback is hard I guess once your OS can't boot.

This is why the client needs have enough error handling to realise it's latest update has now caused unsuccessful boot and roll that update back locally to the last known good configuration (or completely back to factory and pull all updates again).