Thanks, I'll do some testing with these points later on. Though it does sound like go should batch up gethostbyname/getaddrinfo C calls on one or a few threads. Having DNS lookups be the cause of thousands of threads to be created, up till the point where go routines fails is rather ridiculous for a language that's supposed to be good at concurrency imho.
Batching the C calls means that you all lookups are delayed by other lookups. This isn't very good for concurrency especially since it's reliant on network traffic.
If you had a few lookups in the queue, you could end up having goroutines waiting a few seconds or for lots of lookups this could balloon out to minutes for a lookup.
This is the trade off. In general most programs aren't making thousands of concurrent DNS lookups.
Re:Network Programming with Go
by dillonmacc on Monday June 11, 2012 @11:04PM
(#4098484)
i just use http://www.unlocktheinbox.com/dnslookup/mx/ or some other online dns lookup tools.
You can do one of:
1. limit thread creation by using your operating systems limiting features. eg ulimit
2. rebuild the standard lib with DISABLE_CGO=1 to avoid the calls in to C if that's possible in your environment.
3. batch all your dns lookups in to a single goroutine.
4. make changes to the operating system so provide async dns lookups.