I had to read this remark a couple of times before I realised it's suggesting a curl debugging trace is somehow more definitive and less complicated than tcpdump, despite it being a TCP issue.

For those of us who grew up reading W.R.Stevens, that might seem absolutely back-to-front. Not only that, curl ain't gonna help you diagnose the next one, which is a Path MTU Discovery issue, or the one after that, which is an undervoltage on the DRAM refresh controller. It might, however, be helpful in nailing that HTTP/2 prefetch request connection upgrade bug.

Mechanical sympathy means feeling every part of the machine, right down in your bones.

To be fair, it was much simpler to develop and retain fluency in tcpdump when your interfaces were physical and your protocols weren't encrypted.

It is a lot nicer to tcpdump/wireshark when you can see everything, but most of the patterns of bad behavior are visible with encrypted data too.

Of course it's not as easy as it used to be / there's a lot more hurdles.

Encryption as you mentioned; although if you control the client or the server, you can often log keys and decrypt with wireshark, but it's a lot of steps, and you don't get helpful feedback to find mistakes.

NIC offloading means the OS doesn't necessarily see packets as they are on the wire. Segmentation offload means you may see larger packets than are on the wire, and checksum offload often makes sent packets show as errors but they're fine on the wire. If the NIC mutilates the packet, that's hard to debug.

It's not easy to run packet captures on mobile devices. If you can't get tcpdump on the device, you can't get captures from the client side of your cellular data. A lot of people don't have a network router they can run tcpdump on either, and if not, they can't get the client side of wifi data either.

That's the first thing I wanted to do after seeing the JS; since it was a simple request / response, check to see if it's a problem with the client vs the server by replacing the client with a known good piece of code. This effectively bisects the search space without first diving into low-level details.

Exactly. If anyone has ever seen The Price Is Right, this is the optimal strategy for the “higher / lower” game: cut the options in half each time.

Also known as: Binary Search

True, but not as catchy

How would using curl -d make the problem obvious?

It wouldn't have split the request over separate sends.

That's what flushHeaders() is doing; it does a separate send for headers vs the body of the request.

https://nodejs.org/api/http.html#http_request_flushheaders

flushHeaders() is there to have a simple bug to find. In real world cases, tools like tcpdump and strace are essential, and I think this is a nice way to teach about how these tools can be used.

Ok, I think that would have helped me learn that the client can some how trigger the bug but I don't think it would have allowed me to narrow it down to flushHeaders.

There is no such thing as flushing a TCP socket which is why that function jumped out at me. curl would have confirmed the server isn't misbehaving. Googling flushHeaders shows it's a node implementation specific thing and the docs explain that it has to do with the way node buffers http headers. Which makes sense because you don't want to call socket.send() for every http header you add to your request. But you also don't want to send() in between your headers and the body, because then you violate TCP package size expectations.