https://code.google.com/p/go/issues/detail?id=4056
An interesting point raised there is that if they instead used a limited thread pool for all goroutines to share when making OS calls you could produce deadlocks.
That is the (only) mechanism by which a goro can perform such a sync operation without risking blocking the process.
It's entirely possible under some workloads that those blocking OS calls require other goros to make progress (think pipes), so this could result in deadlock (although for some workloads it wouldn't I guess).
I highly doubt that it is creating a thread per goro on client disconnect. If you have a minimalish example of this, the golang mailing list would be very interested in working with you to identify what went wrong and create a patch if it is an issue with the Go implementation.
Close isn't a blocking call, though.
Is there a guarantee on Linux that close can't block? (I suppose it depends on the file type and the definition of blocking.)
I found the (good) result that I could spawn a new goroutine for each incoming connection with minimal (~4k) overhead. This is pretty much what you'd expect since a goro just needs a page for it's stack if it's doing no real work. I had something like 4 VMs each making ~30k conns (from one process) to the central go server with something like 120k conns.
I found one worrying oddity however. Resource usage would spike up on the server when I shut down my client connections (e.g. ctrl-C of a client proc with ~30k conns).
Reasoning about things a bit, I think this is due to the go runtime allocating an OS thread for each goro as it goes through the socket close() blocking call. I think it has to do this to maintain concurrency. So I end up with hundreds of OS threads (each only lives long enough to close(), but I'm doing a lot at the same time).
Can anyone comment:
- is this guess as to the problem likely to be correct?
- is this "thundering herd" a problem in practice?
- are there ways to avoid this? (Other than not using a goro-per-connection, which I think it the only idiomatic way to do it?)
My situation was artificial, but I could well imagine a case that losing, say a reverse proxy, could cause a large number of connections to suddenly want to close() and it would be a shame if that overwhelmed the server.