My money's also on a NIC checksum/segmentation offloading bug. Even though it apparently isn't controllable via the exposed parameters, the card does have these features. "TCP/UDP checksum and segmentation offload (IPv4 and IPv6)" on the spec sheet https://www-ssl.intel.com/content/dam/doc/product-brief/8257...

In the past to rule out NIC problems I have temporarily used a "dumb NIC" like a 3C905 or other NE2000-compatible as a replacement. They're slow because they rely on the host CPU for much of the processing, but a good fallback to use when you want something that is known to work reliably.

Maybe I just haven't hit the right combination of great nic and questionable CPU, but I've seen segmentation offloading result in weird/bad things on the wire, and when I turn it off, I don't see any difference in CPU, so I don't see the point of ever having it on. Specifically for transmit offloading, with dual xeon 2690s, doing https serving via 2x 10gbps nics.

Additionally, I've seen the effects of weird bugs with receive offloading in Linux routers as well: If packets get combined before forwarding, they can be dropped because of the mtu on the next hop interface, so the performance enhancement is actually decreasing performance (and since it's a middle box, I can't do anything except hope my packets don't arrive close enough in time to get aggregated)

Makes sense. Bet it's a Broadcom NIC if it is. Those things are nothing but trouble. I have no idea why HP ship them in their mid-high end kit.