AT&T Fiber in the SF Bay Area is flipping bits

Re:AT&T Fiber in the SF Bay Area is flipping bits by simcop2387 on Monday December 07, 2020 @07:24PM (#25339752)

Take a traceroute from everyone experiencing the problem, look for the common hops among them all. Then compare that list against people not experiancing the problem to find the differences. The finsl set there are good places to start looking. Sweitches and routers along that way could be the cause

[ Parent | on HN ]

Re:AT&T Fiber in the SF Bay Area is flipping bits by grayfaced on Monday December 07, 2020 @07:51PM (#25340041)

If you ping the hops with a large icmp payload, you might be able to observe the flipped bit in the echo reply. That could help isolate which hop it is.

[ Parent | on HN ]

Re:AT&T Fiber in the SF Bay Area is flipping bits by toast0 on Monday December 07, 2020 @07:57PM (#25340103)

You get some amount of your traceroute packet back too, could have flipped bits in there.

[ Parent | on HN ]

Re:AT&T Fiber in the SF Bay Area is flipping bits by toast0 on Monday December 07, 2020 @07:35PM (#25339858)

It might be better (although harder!) to take the traceroute from example.org, instead of from the clients. Forward and reverse paths often diverge, so it's important to find the path with the error.

[ Parent | on HN ]

Re:AT&T Fiber in the SF Bay Area is flipping bits by joshcaron on Monday December 07, 2020 @08:39PM (#25340527)

Some people on Twitter have started collecting IPs: https://twitter.com/alexstamos/status/1336100299841314817

[ Parent | on HN ]

Re:AT&T Fiber in the SF Bay Area is flipping bits by mitchs on Monday December 07, 2020 @10:21PM (#25341342)

They need to be capturing src/dest IPs as well as ports for AT&T to have any hope of using that data.

Edit to make the comment more useful: If anyone is curious, look up "ECMP hashing." There are probably tons of parallel paths through AT&T's network, and to narrow down to the hardware causing problems, they will need to identify which specific path was chosen. Hardware switches packets out equally viable pathways by hashing some of the attributes of the packet. Hash output % number of pathways selects which pathway at every hop.

Hardware does this because everyone wants all packets involved in the same "flow" (all packets with the same src/dest IP and port and protocol (TCP)) to deterministically go through the same set of pipes to avoid packet re-ordering. If you randomly sprayed packets, the various buffer depths of routers (or even speed of light and slightly different length fibers along the way) could cause packets to swap ordering. While TCP "copes" with reordering, it doesn't like it and and older implementations slowed way down when it happened.

[ Parent | on HN ]