Am I the only one who sees a different issue here? The problem is neither stateful service over anycast nor TTL based hashing.

Being a DDoS service, one can imagine the need to have stateful POP since each edge needs to track the TCP state in order to provide DOS protection. At the same time, it is understandable that the state can't be replicated at scale.

As for including TTL in hashing algorithm, it is aimed to solve link under-utilization so it is also a valid implementation.

The real bug here is Arista CPE mangling the TTL bits for the Client Hello packets. I always hate it when networking gear meddles with the protocol stack. Sure it gives some flexibility but time and time again, it ends up breaking something somewhere in the path since much of internet networking is a pile of assumptions. Tampering with protocol fields unilaterally is going to break someone's assumptions somewhere down the path.

Again, there is no hard requirement that all packets in a flow take the same route. Keeping the TCP state machine POP-local when running anycast TCP is exposing a buggy TCP implementation to the Internet. I understand that it is desirable to do so, especially under calm routing conditions where it tends to work, but it is not correct.

I agree that Arista shouldn't be mangling TTL here. However, TTL mangling shouldn't break TCP because TTLs can already change under normal operation, e.g. when routes change, which happens all the time on the Internet. In a non-anycast situation, TCP connections would stay open under these conditions.

To clarify, the Arista routers are the ISP's border routers. It was the residential/business customers' SOHO routers, of various makes and models, that were not decrementing the initial TCP SYN.

Thanks for pointing out. Sorry I overlooked that part. Perhaps I got primed by the opening statements which implied that problem happened only after placing new Arista routers and hastily assumed that it was Arista's routers that mangled the bits.

You are correct, the problem started only after placing new Arista border routers. The previous border routers were not doing ECMP. The issue was a combination of the use of anycast, diverse Internet transit egress, this model of Arista defaulting to using the packet's TTL in its ECMP hash calculations, and the end-customer router CPE egressing packets that are part of the same TCP connection with variable TTL values. Change any one of those items and the issue would not have shown up.