News for hackers. Stuff that matters.
Re:AT&T Fiber in the SF Bay Area is flipping bits by beervirus on Monday December 07, 2020 @08:39PM (#25340526)
Why would a TCP connection allow flipped bits to make it through?
Re:AT&T Fiber in the SF Bay Area is flipping bits by gwittel on Monday December 07, 2020 @09:39PM (#25341000)
Could be crappy/buggy middle boxes. Especially if they’re inspecting packets or messing with SSL.

My ATT modem (Arris bgw-210-700) may have gotten a firmware upgrade recently as I found some settings I shut off got re-enabled. I use it for VDSL but the same model family is used for ATT fiber.

Re:AT&T Fiber in the SF Bay Area is flipping bits by p4coder on Tuesday December 08, 2020 @01:39AM (#25342556)
TCP checksum is simples one's complement checksum. So if two bits are flipped in the same position in two different 16 bit word, they cancel each other out. If you look at all the diffs posted, they differ in even number of lines.
Re:AT&T Fiber in the SF Bay Area is flipping bits by cpach on Monday December 07, 2020 @08:44PM (#25340567)
How do you mean? How would it know if a bit in the HTTP payload got flipped?
Re:AT&T Fiber in the SF Bay Area is flipping bits by sidewndr46 on Monday December 07, 2020 @08:48PM (#25340603)
Each TCP packet has a checksum but it does not catch 100% of all possible bit flips
Re:AT&T Fiber in the SF Bay Area is flipping bits by garmaine on Monday December 07, 2020 @08:51PM (#25340634)
And some routing hardware have been known to ignore this--meaning they'll forward the packet data along and re-calculate the checksum.
Re:AT&T Fiber in the SF Bay Area is flipping bits by tyu2 on Monday December 07, 2020 @09:31PM (#25340948)
Hardware that forwards packets usually forwards IP packets, it doesn't care about TCP checksums and doesn't recalculate them. TCP stack in your OS is the one that does that.

It has to be a special kind of hardware that does much deeper packet inspection (DPI) to recalculate TCP checksums, usually used for spying, throttling, censorship, injecting ads, injecting exploits, etc., but not merely routing/forwarding packets.

Re:AT&T Fiber in the SF Bay Area is flipping bits by pclmulqdq on Monday December 07, 2020 @11:03PM (#25341620)
Switches (L2 devices) recalculate Ethernet CRCs (L2 error detection codes), and routers tend to calculate TCP/UDP checksums (L3 error detection codes) and everything below. I've seen exactly this issue with switches that have bad RAM before, and I assume that they just have a router with bad RAM (and without ECC RAM, which looks embarrassing).
Re:AT&T Fiber in the SF Bay Area is flipping bits by noncoml on Monday December 07, 2020 @11:49PM (#25341973)
To my knowledge, from working on an actual software router, a router will only touch the TTL and recalculate the IP header checksum.

There is no reason for it to touch the TCP header.

Re:AT&T Fiber in the SF Bay Area is flipping bits by garmaine on Tuesday December 08, 2020 @12:21AM (#25342196)
Agreed there is no logical reason for it to touch the TCP header.

And yet, an unfortunate number of L2 switches do exactly that. :(

Re:AT&T Fiber in the SF Bay Area is flipping bits by Thorrez on Tuesday December 08, 2020 @01:10AM (#25342420)
Doesn't NAT (specifically carrier-grade NAT in this case) often modify the port? Although I don't know if AT&T does carrier-grade NAT.

https://en.wikipedia.org/wiki/Carrier-grade_NAT

Re:AT&T Fiber in the SF Bay Area is flipping bits by noncoml on Tuesday December 08, 2020 @01:26AM (#25342490)
> an unfortunate number of L2 switches do exactly that

Can you name any? Just curious

Re:AT&T Fiber in the SF Bay Area is flipping bits by beervirus on Monday December 07, 2020 @08:51PM (#25340630)
Checksums. TCP is supposed to provide reliability against data corruption, resend bad packets, etc.
Re:AT&T Fiber in the SF Bay Area is flipping bits by mitchs on Monday December 07, 2020 @10:18PM (#25341318)
Unfortunately TCP checksums are hot garbage given switch ASIC design. They are a 16 bit one's complement sum over a packet. If you get two bit flips in the same offset % 16, you can pass a checksum.

The problem is routers slow down the high speed serial signals from fiber to by splitting the bits over a large number of slower speed signals internally. Often those wider busses are a multiple of 16 bits. For example, one ASIC I know of moves things around in 204 byte chunks. (Might have been 208, been a while.) Anyway, the problem is that if there is a defect in one of those parallel elements it will always flip bits in the same offset position mod 204 bytes, which is the same position mod 16 bits. If the hardware is degraded enough, it can end up flipping two bits in the same position, and that has a fairly good chance of passing the checksum.

Ethernet has proper CRCs on packets, which is a lot less vulnerable to shenanigans like this, but unfortunately those can end up being checked on the way in, discarded, and then re-generated on the way out of a router. If anything is corrupted in the middle of the switch ASIC, nothing notices and it passes along. I once helped troubleshoot an issue in our network where a BGP packet was corrupted in this way. The flipped bits ended up causing a more specific route to be generated, and we had the world weirdest BGP route hijack within the bounds of our own data center.

Re:AT&T Fiber in the SF Bay Area is flipping bits by myrandomcomment on Tuesday December 08, 2020 @01:44AM (#25342584)
204 IIRC. Dune Petra.
Re:AT&T Fiber in the SF Bay Area is flipping bits by rob-olmos on Monday December 07, 2020 @09:47PM (#25341054)
Depending on TCP checksums and experiencing single-bit flips took down AWS S3 back in the day: https://status.aws.amazon.com/s3-20080720.html
Re:AT&T Fiber in the SF Bay Area is flipping bits by thamer on Monday December 07, 2020 @09:41PM (#25341013)
Checksums are often calculated in hardware on the NIC. I have personally seen a network card send packets with corrupted data (corrupted by the network card itself) and valid TCP checksums, computed on the corrupted data.
Re:AT&T Fiber in the SF Bay Area is flipping bits by tyu2 on Monday December 07, 2020 @09:57PM (#25341147)
That's true, but doesn't apply in this case if you assume you received one with wrong checksum, a network card won't recompute it of course.