Re:Summary of June 8 outage
by unityByFreedom on Wednesday June 09, 2021 @05:44AM
(#27446052)
What are Europe's whales?
[ Parent
| on HN ]
Lots of tech companies have engineering offices with SRE responsabilities in Europe timezones. Could be anyone really.
Re:Summary of June 8 outage
by unityByFreedom on Wednesday June 09, 2021 @06:43AM
(#27446391)
My guess is this level of change would come from senior devs in the main office.
Are you implying that no senior devs or main offices are located in europe?
Also, just because the rollout to fastly happened at a morning EU time, doesn't mean that the change was made. If there's a deployment pipeline, it could have been made 2-3 hours earlier, or even the day before
Re:Summary of June 8 outage
by unityByFreedom on Wednesday June 09, 2021 @06:51AM
(#27446455)
I'm saying a big network change is probably going to be executed while HQ is awake and in the loop. In my experience the most skillful devs aren't the most senior.
We don't know much about the specific client configuration change that triggered this condition yet. It doesn't necessarily have to be a big company wide infrastructure change.
To me it sounds plausible that an SRE team in an alternate location made a change scoped to their permission level, following company-directed playbooks, which eventually triggered the faulty condition at Fastly.
Re:Summary of June 8 outage
by unityByFreedom on Wednesday June 09, 2021 @09:26AM
(#27447590)
Seems likely to me this was a change pushed from outside the US given the timing.
Many big tech companies (and individual products within the company) rotate oncall/release/SRE responsibilities between NA and EU. HQ doesn't need to be awake in these cases.
Re:Summary of June 8 outage
by iainmerrick on Wednesday June 09, 2021 @04:31PM
(#27452815)
Spotify is the first that comes to mind, if you’re looking for a big tech company headquartered in Europe.