I was under the impression there was enough layers of checksums once you get to the level of UDP/TCP that these kind of single bit flips should be detected and dropped before you get to read them. What's going on here? Is networking gear not calculating checksums like it should?

TCP header does have a checksum that is supposed to check both the header and the payload.

So either the bit corruption is such that is not detected by the checksum

Or ATT is doing something nefarious and touches layer 4 and corrupts the data while doing so

I don’t have numbers off hand, but from a feeling I have from memory, I would think it is extremely unlikely that TCP checksums are consistently failing to trigger retransmission. Someone must be altering packets along the way.

TCP checksums are notoriously weak:

https://www.evanjones.ca/tcp-checksums.html

With enough packets passing the dodgy RAM a noticeable number will manage to get mangled in such a way that the checksum is still correct.

Checksums are also often recomputed on transit if the packet is intercepted, e.g. to limit TTL, unflag odd/unused TCP features, that kind of ISP-ish preening. So if it was a software error or even a hardware problem in the right (wrong) spot it’s possible to get this kind of corruption without retransmits.

I think you are confusing the IP header with TCP header.

Routers don’t touch the TCP header at all

No. URG, “Christmas tree” packets, etc. all can be mangled by the ISP. Or more commonly dropped

if your router does NAT it touches the tcp header

You are right, although NAT could be classified as a firewall feature, that runs on routers.

But in any case it's irrelevant in this context as ATT shouldn't be doing any NAT

In that case, won't there be significant packet loss causing throughput to be very slow? I don't know if this is possible without something messing with TCP headers.

It is triggering retransmission. It just retransmits until it gets lucky with the (pretty weak) TCP checksum.

The checksum should be checked by your computer. So somehow the packet is being repackaged with the correct checksum, but for the wrong data. In other words, when your computer checks the checksum, it matches. Another possibility is that somehow only errors that result in the same checksum are being generated.

Or their computer isn't checking the checksum. As is apparently the case on mac os (as reported elsewhere in the thread).

Another quality product by Apple ;P

Except it was trivial to reproduce with the script on non-Apple devices, and people in one of the many Twitter threads surrounding this showed that on their Mac there was MANY tcp retransmits due to invalid checksums, and the bit-flipped packet did have the correct checksum.

https://twitter.com/KielO/status/1335640544496689152

OK, that tweet does show the checksum is OK. I didn't see a whole lot of tcpdumps, so had to go with what was reported in the thread (I tried to reproduce with a few people, but my server wasn't in the broken path, so I couldn't get a lot of real data).

That tweet in particular doesn't show any retransmits.

tcpdump/wireshark gets a little hard to read at times; especially when the packet dump is a lie: all those packets marked red for bad checksums are from the dumping machine, and the checksums are wrong because the NIC is filling them in, and the capture interface doesn't get to see what they are). Perhaps the other people in the thread who said mac os was ignoring bad checksums were also confused; or perhaps it does ignore bad checksums, it's pretty bad at networking (it can't handle a synflood in 2020 because it's got synhandling code from 2000)

There’s no proof of this. And what possible reason would macOS have for not checking the checksum? Although the checksum is weak it presumably catches at least some corrupt traffic. Do you really think Apple would just skip the TCP checksum and make its network performance less reliable when they have already implemented (or maintained if it came from BSD) the rest of a TCP/IP stack, which is vastly more complex, just because its developers are lazy?

There were reports in the twitter threads that macs were ignoring the bad checksum (and I thought I saw confirmations in this thread, blaming a driver). Since I don't have a mac anymore, I couldn't confirm or deny; and since I don't have any equipment with the bad equipment in the path, I couldn't get a tcpdump, and I hadn't seen any on twitter.

I wouldn't have expected Apple to purposefully break the checksum, just like they I don't think they purposefully have no synflood protection because they pulled the TCP/IP stack at the turn of the century and never pulled it to get the many many many upgrades from upstream (although, they did add on MP-TCP, so there's that). I wouldn't be surprised if tcp checksums had stopped working ages ago, possibly because of a aggressive driver, and nobody noticed. Kind of like how if you spawn a few thousand threads that just sit around sleeping, it will delay watchdog kicks and the kernel will panic. (also, from reports on here, not personal experience)

It also seems quite easy to verify this hypothesis with scapy:

  >>> p1 = IP(dst="192.168.mac.ip")/TCP(dport=1984,sport=20001)
  >>> p2 = IP(dst="192.168.mac.ip")/TCP(dport=1984,sport=20002)
  >>> p2.show2()
  ###[ IP ]###
    version   = 4
    ihl       = 5
    tos       = 0x0
    len       = 40
    id        = 1
    flags     =
    frag      = 0
    ttl       = 64
    proto     = tcp
    chksum    = 0xf905
    src       = 192.168.linux.ip
    dst       = 192.168.mac.ip
    \options   \
  ###[ TCP ]###
       sport     = commtact_http
       dport     = bb
       seq       = 0
       ack       = 0
       dataofs   = 5
       reserved  = 0
       flags     = S
       window    = 8192
       chksum    = 0xb836
       urgptr    = 0
       options   = []

  >>> p2[TCP].chksum = 0xb836 ^ 0x8  # mangled checksum

  >>> sr1(p1, timeout=1)
  Begin emission:
  Finished sending 1 packets.
  ....*
  Received 5 packets, got 1 answers, remaining 0 packets
  <IP  version=4 ihl=5 tos=0x0 len=44 id=0 flags=DF frag=0 ttl=64 proto=tcp chksum=0xb902 src=192.168.mac.ip dst=192.168.linux.ip |<TCP  sport=bb dport=microsan seq=1671494800 ack=1 dataofs=6 reserved=0 flags=SA window=65535 chksum=0x6039 urgptr=0 options=[('MSS', 1460)] |>>

  >>> sr1(p2, timeout=1)
  Begin emission:
  Finished sending 1 packets.
  .....................................................................................................................................................................................
  .....................................................................................................................................................................................
  ....................................................................................................................................
  Received 494 packets, got 0 answers, remaining 1 packets

My Mac silently dropped the packet with the mangled checksum.

It is extremely common for hardware to be configured to ignore checksums. ("A packet with a bad checksum would have been dropped before it got here. Our cabling is too short to drop bits.")

The same here. It is always the 0x08 bit. (p-x, d-l, g-o etc.).

I wonder if there’s some 64 line I/O somewhere with a couple traces bridged together, where:

0 and 0 are still 0 and 0,

1 and 1 are still 1 and 1,

but 0 and 1 become 1 and 1 (or 0&0)

and 1 and 0 become 1 and 1 (or 0&0).

I remember seeing this wackiness when I bridged two address lines on an EEPROM with the tiniest amount of solder.

RoHS/tin whiskers strikes again?

This is unlikely, given how many people are seeing this. And now the problem has gone away for me!

I get the same. Pretty amazing that the only effect from such garbled data that I noticed were some annoying hangs and bad Twitter links!