It's likely this is related to DNS lookups.
Doing a lookup on a modern operating system requires calling in to some C library. The call in to the C code means that the thread can no longer be used to schedule goroutines, the runtime doesn't know how long the C call will take so it creates a new thread to keep running goroutines.
For calls in to C that are very quick this can lead to a lot of thread creation.
You can do one of:
1. limit thread creation by using your operating systems limiting features. eg ulimit
2. rebuild the standard lib with DISABLE_CGO=1 to avoid the calls in to C if that's possible in your environment.
3. batch all your dns lookups in to a single goroutine.
4. make changes to the operating system so provide async dns lookups.
Thanks, I'll do some testing with these points later on. Though it does sound like go should batch up gethostbyname/getaddrinfo C calls on one or a few threads. Having DNS lookups be the cause of thousands of threads to be created, up till the point where go routines fails is rather ridiculous for a language that's supposed to be good at concurrency imho.
Batching the C calls means that you all lookups are delayed by other lookups. This isn't very good for concurrency especially since it's reliant on network traffic.
If you had a few lookups in the queue, you could end up having goroutines waiting a few seconds or for lots of lookups this could balloon out to minutes for a lookup.
This is the trade off. In general most programs aren't making thousands of concurrent DNS lookups.
Re:Network Programming with Go
by dillonmacc on Monday June 11, 2012 @11:04PM
(#4098484)
i just use http://www.unlocktheinbox.com/dnslookup/mx/ or some other online dns lookup tools.
It was quite disturbing that firing off several thousand goroutines resulted in several thousand native threads - resulting the system trashing. If I inserted a very small pause between creating the go routines, this did not occur and just 1 or 2 native threads were created. I tried the runtime.GOMAXPROCS, to no awail. Is there any way of controlling this kind of behavior ?