Re:Summary of June 8 outage
by unityByFreedom on Wednesday June 09, 2021 @06:51AM
(#27446455)
I'm saying a big network change is probably going to be executed while HQ is awake and in the loop. In my experience the most skillful devs aren't the most senior.
We don't know much about the specific client configuration change that triggered this condition yet. It doesn't necessarily have to be a big company wide infrastructure change.
To me it sounds plausible that an SRE team in an alternate location made a change scoped to their permission level, following company-directed playbooks, which eventually triggered the faulty condition at Fastly.
Re:Summary of June 8 outage
by unityByFreedom on Wednesday June 09, 2021 @09:26AM
(#27447590)
Seems likely to me this was a change pushed from outside the US given the timing.
Also, just because the rollout to fastly happened at a morning EU time, doesn't mean that the change was made. If there's a deployment pipeline, it could have been made 2-3 hours earlier, or even the day before