lacop

That seems to work, thank you!

Now latency is just RTT + server time + payloadsize/bandwidth, not multiple times RTT: https://github.com/grpc/grpc-go/issues/8436#issuecomment-311...

I was not aware of this setting, it's pretty unfortunate this is a system-level setting that can't be overridden on application layer, and the idle timeout can't be changed either. Will have to figure out how to safely make this change on the k8s service this is affecting...

Yeah that was my understanding too, hence I filed the bug (actually duplicate of older bug that was closed because poster didn't provide reproduction).

Still not sure if this is linux network configuration issue or grpc issue, but something is for sure broken if I can't send a ~1MB request and get response within roughly network RTT + server processing time.

Doesn't initcwnd only apply as the initial value? I don't care that the first request on the gRPC channel is slow, but subsequent requests on the same channel reuse the TCP connection and should have larger window size. This works as long as the channel is actively being used, but after short inactivity (few hundred ms, unsure exactly) something appears to revert back.

Somewhat related, I'm running into a gRPC latency issue in https://github.com/grpc/grpc-go/issues/8436

If request payload exceeds certain size the response latency goes from network RTT to double that, or triple.

Definitely something wrong with either TCP or HTTP/2 windowing as it doesn't send the full request without getting ACK from server first. But none of the gRPC windowing config options nor linux tcp_wmem/rmem settings work. Sending one byte request every few hundred milliseconds fixes it by keeping the gRPC channel / TCP connection active. Nagle / slow start is disabled.

I got some empirical data on this!

Effingo file copy service does application-layer strong checksums and detects about 4.5 corruptions per exabyte transferred (figure 9, section 6.2 in [1]).

This is on top of TCP checksums, transport layer checksums/encryption (gRPC), ECC RAM and other layers along the way.

Many of these could be traced back to a "broken" machine that was eventually taken out.

[1] https://dl.acm.org/doi/abs/10.1145/3651890.3672262

Unfortunately I don't think browsers handle opening MHTML well out of the box.

At least last time I checked in only worked for local paths (file://) and only some browsers. Otherwise it would either try to download or just show plain text.

I ended up using Chrome to dump to MHTML [1] and then reshuffle the content into individual files, rewriting the path references and fixing mime types [2]. That gives a reasonably faithful static capture of a page that can be shared as link.

[1] https://chromedevtools.github.io/devtools-protocol/tot/Page/... [2] https://github.com/lacop/udrb/blob/master/app/src/chrome/mod...