Wow. Can you easily change the tcp connection timeout?

You can. It’s trivial once you know it’s possible. Not sure why it’s not set by default. https://go.googlesource.com/net/+/master/http2/transport.go

To be clear, this is for http/2, not tcp. You can very easily set read and write deadlines on tcp conns, but you can’t detect if a peer has disappeared without data. You can set keepalive but it’s not reliable and varies wildly between OSs.

You need a heartbeat or ping message together with an advancing deadline to detect dead peers reliably.

HTTP/2 supports a heartbeat in the protocol using PING frames. But I guess a lot of clients probably don’t support it or use it by default.

that sounds like there is pooling going on and not invalidating the pooled connection when a timeout happens. I've actually seen a lot of libraries in other languages do a similar thing (my experience is some of the elixir libraries don't have good pool invalidation for http connections). having a default invalidation policy that handles all situations is a bit difficult but I think a default policy that invalidates on any timeout is much better than a default policy that never invalidates on a timeout. as long as invalidation means just evicting it from the pool and not tearing down other channels on the HTTP/2 connection. for example you could have a timeout on a HTTP/2 connection that is just on an individual channel but there is still data flowing through the other channels.

You can configure the HTTP/2 client to use a timeout + heartbeat.

https://go.googlesource.com/net/+/master/http2/transport.go

That's a big file. Mind pointing to a specific line number?