Arista's default of having the TTL as part of the hashing function to calculate packet path means that any TCP connection that does not have a consistent TTL when it reaches this (source) ISP's routers can potentially be bucketed onto multiple different egress paths (transit or peering connections) onto different upstream SP's.

Because BGP works on the concept of AS Numbers rather than IP's (and because Fastly uses Anycast, which announces the same set of IP's from multiple POP's), it's possible that packets that leave the ISP network over different transit connections end up at different Fastly POP's, as each transit provider or peer will have a different route to Fastly's AS.

Fastly does not share TCP session information between POP's as I imagine at that scale it's massively prohibitive to do so.

The TCP RST happens because there is no established TCP session at the secondary POP which receives the TLS handshake packet, and because TTL is part of the hash function it will always end up following a different path (and therefore hitting a different datacenter, if routing stays consistent) if the TTL is reduced by 1.

It's generally deemed 'acceptable' in Anycast terms to cause a TCP RST when traffic switches from one POP to another, as this usually happens rarely, in response to changes in the path between two AS numbers.

For things like websites, browsers will usually just retry the connection, and as the path has changed and stays consistent, it starts to work and all you notice is a slight delay.

It doesn't work in this situation because the second packet with the reduced TTL is consistently routed to an 'incorrect' datacenter and breaks the TCP session every time, effectively dropping service to 0.

>For things like websites, browsers will usually just retry the connection

This is not the case. Every single TCP RST you send on an active http connection will lead to an http request failing in a way which causes application brokenness. No browser auto-retries in that case. The user always has to hit refresh to fix it.

Of course that kind of routing would create a potential bottleneck for an attacker to exploit ("simply" force traffic to the wrong IPs to the wrong PoP, assuming $attacker had this level of access to the backbone), but that's the problem Fastly are supposedly paid to deal with

Their scheme is fine and dandy with a protocol like DNS where UDP retries are transparent and TCP is a tiny fraction of weird traffic, but for business applications handling credit cards, surely the occasional RST is already too many

Or another way to look at it, basically they're saying their IP addresses are special snowflakes and actually the full address includes the route, and source networks are wrong for assuming things work the way they're supposed to everywhere else on the Internet

It's actually simpler than that — the state of the TCP connection is controlled by the hosts to the TCP connection, so, all it takes is for a "client" host to pretend that the connection has already been established (sending a single packet alleging as such) — no need for any special access to any backbone.

please cite some sources to back this up. i find this statement surprising.

Like, an application which implemented "stateless" HTTP endpoints would still run into this issue?