News for hackers. Stuff that matters.
Re:Adventures in debugging: etcd, HTTP pipelining, and file ... by philips on Thursday November 05, 2015 @04:58AM (#10512529)
This is some interesting behavior from the blog post:

    Eventually, Python exits, and closes TCP connection A,
    by sending a FIN packet – but etcd doesn’t send a FIN
    of its own, or send a zero-byte chunk to finish the 200 
    OK, as it does in the success case. This causes the 
    socket to leak on etcd’s side – we think this is a bug
    in Go’s HTTP stack.
This really should work so we need to investigate that one more deeply. I am not opposed to adding the timeout option as the author would like to do and we would happily consider a patch for this.

Update: here is a proof-of-concept patch to address this feature request: https://github.com/coreos/etcd/pull/3816

Longer term etcd will use gRPC/http 2.0 in the new etcd v3 API[1] to address use cases like this. In particular gRPC will allow us to have multiple streams while keeping open a single TLS connection. A major expense of the current watch API is having to tear down connections as explained in this post. We will still support http/rest likely through the use of grpc-gateway[2].

The file descriptor rlimit issue that was the side-effect of this issues is why databases like etcd have to protect themselves. For example a few months ago etcd learned to reserve file descriptors for itself in case we get clients hogging up resources[2]. Longer term we would like to be able to let proxies handle watches directly to further protect the core quorum of etcd.

[1] https://github.com/coreos/etcd/blob/master/Documentation/rfc... [2] https://github.com/gengo/grpc-gateway [3] https://github.com/coreos/etcd/pull/3219

Re:Adventures in debugging: etcd, HTTP pipelining, and file ... by noselasd on Thursday November 05, 2015 @03:57PM (#10516062)
I don't think it's a bug in Go's http stack. The TCP FIN from the client just says "I've closed my write end of the connection". In this case the client actually did a full close of the socket, i.e. it's actually closed for both reading and writing - but TCP doesn't communicate that fact. And HTTP says that in this situation (the other end closed its write half of the connection), you should still send the response back and then close your end of the connection.

The server can't know[1] that client has fully closed the connection until it tries to write something to it - I'm guessing there is some sort of long polling going on here, so the server has parked the http request somewhere and isn't going to write anything to the connection until something happens (if ever). After all, at this point the server doesn't know the TCP connection is fully closed, it thinks it's half closed.

[1] Unless you also enable tcp keepalive, the keepalive would discover that the socket is closed on the other end (after a couple of hours by default).

Re:Adventures in debugging: etcd, HTTP pipelining, and file ... by mkulke on Thursday November 05, 2015 @08:00AM (#10513027)
We have a very nasty issue in kubernetes with it's userspace-proxy leaking handles, when misbehaving workload doesn't close connections properly (e.g. Java InputStreams). Could this be related?
Re:Adventures in debugging: etcd, HTTP pipelining, and file ... by philips on Thursday November 05, 2015 @08:11AM (#10513076)
Maybe. The kubernetes 1.0 proxy is a pure TCP proxy so would need more details. Could you file an issue?

Kubernetes 1.1 will have an iptables based proxy too: https://github.com/kubernetes/kubernetes/issues/3760#issueco...

Re:Adventures in debugging: etcd, HTTP pipelining, and file ... by mkulke on Thursday November 05, 2015 @09:24AM (#10513422)
There is an open issue in which we came to more or less the same conclusion as mentioned the article (not a bug, but a feature of the TCP/IP protocol).

i am a bit puzzled why other people are not constantly bitten by this, though.