News for hackers. Stuff that matters.
Re:Summary of June 8 outage by hulitu on Wednesday June 09, 2021 @05:16AM (#27445888)
They forgot to test it.
Re:Summary of June 8 outage by alkonaut on Wednesday June 09, 2021 @06:05AM (#27446183)
Test “it”? The change in question wasn’t by fastly but a customer of theirs making a config change. It’s possible that this customer did validate their change somehow.

Fastly obviously didn’t test their code (with the bug) enough, but testing of course can never prove the absence of bugs. Testing for a global deployment like a massive CDN happens to a large extent in prod because you don’t have another globe. You can test on a smaller scale but eventually you run into a problem that only shows itself at full scale.

Re:Summary of June 8 outage by that_guy_iain on Wednesday June 09, 2021 @06:19AM (#27446264)
Fastly even say it was a valid change.

> We experienced a global outage due to an undiscovered software bug that surfaced on June 8 when it was triggered by a valid customer configuration change.

in the first sentence

Re:Summary of June 8 outage by alkonaut on Wednesday June 09, 2021 @06:27AM (#27446309)
Their change was bad, that was May 12. Since that seemed OK on May 13,14,… there wasn’t much indicating that change would blow up weeks later. For example if they roll it out gradually, they would reach 100% rollout with all lights being green

The customer change was a valid configuration. That was yesterday.

Re:Summary of June 8 outage by 2rsf on Wednesday June 09, 2021 @05:59AM (#27446154)
Testing is never complete nor can it be theoretically complete