News for hackers. Stuff that matters.
Re:Summary of June 8 outage by unityByFreedom on Wednesday June 09, 2021 @06:43AM (#27446391)
My guess is this level of change would come from senior devs in the main office.
Re:Summary of June 8 outage by maccard on Wednesday June 09, 2021 @06:48AM (#27446431)
Are you implying that no senior devs or main offices are located in europe?

Also, just because the rollout to fastly happened at a morning EU time, doesn't mean that the change was made. If there's a deployment pipeline, it could have been made 2-3 hours earlier, or even the day before

Re:Summary of June 8 outage by unityByFreedom on Wednesday June 09, 2021 @06:51AM (#27446455)
I'm saying a big network change is probably going to be executed while HQ is awake and in the loop. In my experience the most skillful devs aren't the most senior.
Re:Summary of June 8 outage by vbsteven on Wednesday June 09, 2021 @07:14AM (#27446579)
We don't know much about the specific client configuration change that triggered this condition yet. It doesn't necessarily have to be a big company wide infrastructure change.

To me it sounds plausible that an SRE team in an alternate location made a change scoped to their permission level, following company-directed playbooks, which eventually triggered the faulty condition at Fastly.

Re:Summary of June 8 outage by unityByFreedom on Wednesday June 09, 2021 @09:26AM (#27447590)
Seems likely to me this was a change pushed from outside the US given the timing.
Re:Summary of June 8 outage by yupper32 on Wednesday June 09, 2021 @04:48PM (#27453045)
Many big tech companies (and individual products within the company) rotate oncall/release/SRE responsibilities between NA and EU. HQ doesn't need to be awake in these cases.