News for hackers. Stuff that matters.
Re:Summary of June 8 outage by EE84M3i on Wednesday June 09, 2021 @12:20AM (#27444187)
Absolutely no details about the bug or why a single customer configuration effected global state on the server, or why this wasn't caught by configuration change safety mechanisms/smoke tests/gradual rollout.

Also, what is up with their partitioning? Do they seriously have one customer that gets served from 85% of their servers? Is it a whale?

Good on them for getting a statement out right away (although they basically had to) but seems to be lacking any useful details. Wonder if they were scrubbed by PR/legal in hopes of reducing the number of customers coming to ask for gibs.

Re:Summary of June 8 outage by abestic9 on Wednesday June 09, 2021 @12:43AM (#27444289)
It's an update on the situation until they confirm the bug fix is completely rolled out. I certainly wouldn't expect they tell their customer base how they can exploit a bug to bring down part of their business.

With respect to partitioning - we don't know how or why an invalid configuration could poison so many nodes; if the config was physically present on them or if there was a cascade of healing/balancing issues stemming from it.

I would leave speculation on many of your points at the doorstep until we see a full report.

Re:Summary of June 8 outage by plasma on Wednesday June 09, 2021 @12:52AM (#27444333)
At a guess, perhaps each server can serve everyone, and the system-wide config file that is shipped to each server (to handle each customer) perhaps became corrupted as part of a customer update, and it was rolled out to every node assumed to be correct.

A defense against this could be to ensure the system that applies the change validates some health-checks continue to work after the new file is made (or automatically rollback to previous configuration).

I can see how this would happen, assuming thats what happened.

Re:Summary of June 8 outage by EE84M3i on Wednesday June 09, 2021 @12:59AM (#27444376)
Yes, it sounds sort of like that. But this can be remediated by two things I was asking about: smoke tests and partitioning.

When making a config change I'd assume they don't make it to all servers at once and instead roll it out gradually. If this caused the server to instantly start 503'ing all customers, presumably this would have been caught - perhaps it was more delayed though (resource leak, etc) and obviously that is somewhat more difficult to catch.

If they're properly partitioning customers, ideally they wouldn't even ship the configs to all servers (slightly less good, but still pretty good they could ship them there but not parse/load them). It sounds like at the least this customer's config change effected 85% of servers, which seems absurd to me.

So yes, I can see how it happened, but for Fastly, which runs one of the biggest CDNs, these don't seem like very reasonable mistakes.

Re:Summary of June 8 outage by sersi on Wednesday June 09, 2021 @04:41AM (#27445621)
One of the big competitive advantage Fastly has compared to say Akamai is that configuration changes roll out extremely fast. I could see them skimping on smoke tests to keep that advantage and not thinking that this could ever happen.
Re:Summary of June 8 outage by notyourday on Wednesday June 09, 2021 @05:44AM (#27446057)
At its core Fastly uses heavily modified Varnish 2 with customer's having full access to vcl providing an unprecedented level of control of the request processing and cache behavior. It is extremely difficult, if not impossible, to completely remove foot-guns while retaining this ability. They do amazing thing checking validity of the code and aborting broken requests but it is still vcl and in its core varnish 2 does not have multi-tenancy segmentation.
Re:Summary of June 8 outage by notacoward on Wednesday June 09, 2021 @07:59AM (#27446884)
Thank you for that. That last clause seems like the real key here. With that much power in customers' hands and no hard multi-tenant isolation, unintentional DoS becomes almost inevitable. In effect, it puts every customer at the mercy of every other's diligence (or lack thereof). Even with diligence, that seems a bit fragile.
Re:Summary of June 8 outage by cowsandmilk on Wednesday June 09, 2021 @06:21AM (#27446276)
> one customer that gets served from 85% of their servers

There could be a feedback loop that is the opposite of a smoke test.

1. Validate customer configuration, if passes, assume it can roll out 2. Roll out customer configuration to node 3. Node goes down 4. Migrate all customers on node to new nodes 5. Node that problematic customer was migrated to goes down 6. Rinse and repeat as problematic customer migrates to every node and takes out every last one.

Re:Summary of June 8 outage by alkonaut on Wednesday June 09, 2021 @06:08AM (#27446197)
Smoke tests or gradual rollout won’t help with the change by fastly on May 12 when they deployed the buggy change. It’s likely that this was gradually rolled out and looked ok with all customer configs existing on May 12. Obviously there should be other safeguards in place but gradual rollout in itself wouldn’t have helped since it would look green at a 100% rollout weeks ago.
Re:Summary of June 8 outage by EE84M3i on Wednesday June 09, 2021 @06:13AM (#27446223)
I meant smoke testing and gradual rollout of the config, not the code, although obviously that's important too.
Re:Summary of June 8 outage by alkonaut on Wednesday June 09, 2021 @06:27AM (#27446307)
You’d think that individual customer configuration changes should only ever affect that customer, and that gradual vs instant rollout would be an option the customer handles when changing their configuration!
Re:Summary of June 8 outage by unityByFreedom on Wednesday June 09, 2021 @03:51AM (#27445342)
> Is it a whale?

TikTok is my guess. ByteDance is valued at 250 billion. Plus, the change was pushed in the middle of the night, which would be daytime in Asia. Certainly there are other development teams in Asia, but considering the scale of the change it likely comes from HQ, and Fastly's whale in Asia would be them.

edit They may have lost TikTok at the end of last year, either partially or completely [1]. Anyone know what they use now? Akamai, or maybe they stealthily switched back to Fastly?

[1] https://www.barrons.com/articles/fastly-lost-tiktok-but-ceo-...

Re:Summary of June 8 outage by Goz3rr on Wednesday June 09, 2021 @05:41AM (#27446027)
Middle of the night? It was at noon for most of Europe
Re:Summary of June 8 outage by unityByFreedom on Wednesday June 09, 2021 @05:44AM (#27446052)
What are Europe's whales?
Re:Summary of June 8 outage by vbsteven on Wednesday June 09, 2021 @05:58AM (#27446146)
Lots of tech companies have engineering offices with SRE responsabilities in Europe timezones. Could be anyone really.
Re:Summary of June 8 outage by unityByFreedom on Wednesday June 09, 2021 @06:43AM (#27446391)
My guess is this level of change would come from senior devs in the main office.
Re:Summary of June 8 outage by maccard on Wednesday June 09, 2021 @06:48AM (#27446431)
Are you implying that no senior devs or main offices are located in europe?

Also, just because the rollout to fastly happened at a morning EU time, doesn't mean that the change was made. If there's a deployment pipeline, it could have been made 2-3 hours earlier, or even the day before

Re:Summary of June 8 outage by unityByFreedom on Wednesday June 09, 2021 @06:51AM (#27446455)
I'm saying a big network change is probably going to be executed while HQ is awake and in the loop. In my experience the most skillful devs aren't the most senior.
Re:Summary of June 8 outage by vbsteven on Wednesday June 09, 2021 @07:14AM (#27446579)
We don't know much about the specific client configuration change that triggered this condition yet. It doesn't necessarily have to be a big company wide infrastructure change.

To me it sounds plausible that an SRE team in an alternate location made a change scoped to their permission level, following company-directed playbooks, which eventually triggered the faulty condition at Fastly.

Re:Summary of June 8 outage by unityByFreedom on Wednesday June 09, 2021 @09:26AM (#27447590)
Seems likely to me this was a change pushed from outside the US given the timing.
Re:Summary of June 8 outage by yupper32 on Wednesday June 09, 2021 @04:48PM (#27453045)
Many big tech companies (and individual products within the company) rotate oncall/release/SRE responsibilities between NA and EU. HQ doesn't need to be awake in these cases.
Re:Summary of June 8 outage by iainmerrick on Wednesday June 09, 2021 @04:31PM (#27452815)
Spotify is the first that comes to mind, if you’re looking for a big tech company headquartered in Europe.
Re:Summary of June 8 outage by davman on Wednesday June 09, 2021 @06:52AM (#27446462)
gov.uk ?
Re:Summary of June 8 outage by Sander_Marechal on Wednesday June 09, 2021 @05:28AM (#27445958)
My guess is reddit. It was down yesterday due to this.
Re:Summary of June 8 outage by unityByFreedom on Wednesday June 09, 2021 @05:41AM (#27446033)
Why would reddit push such a change at 4am SF time? It's also unlikely that a high load at that time would trigger anything. Reddit's peak activity times are US daylight hours.
Re:Summary of June 8 outage by fierro on Wednesday June 09, 2021 @11:51AM (#27449329)
guessing it's somehow BGP related. can't wait to find out