News for hackers. Stuff that matters.
Re:Network Programming with Go by jesstaa on Monday June 11, 2012 @05:51AM (#4093859)
It's likely this is related to DNS lookups. Doing a lookup on a modern operating system requires calling in to some C library. The call in to the C code means that the thread can no longer be used to schedule goroutines, the runtime doesn't know how long the C call will take so it creates a new thread to keep running goroutines. For calls in to C that are very quick this can lead to a lot of thread creation.

You can do one of:

1. limit thread creation by using your operating systems limiting features. eg ulimit

2. rebuild the standard lib with DISABLE_CGO=1 to avoid the calls in to C if that's possible in your environment.

3. batch all your dns lookups in to a single goroutine.

4. make changes to the operating system so provide async dns lookups.

Re:Network Programming with Go by noselasd on Monday June 11, 2012 @08:42AM (#4094277)
Thanks, I'll do some testing with these points later on. Though it does sound like go should batch up gethostbyname/getaddrinfo C calls on one or a few threads. Having DNS lookups be the cause of thousands of threads to be created, up till the point where go routines fails is rather ridiculous for a language that's supposed to be good at concurrency imho.
Re:Network Programming with Go by jesstaa on Monday June 11, 2012 @10:10AM (#4094646)
Batching the C calls means that you all lookups are delayed by other lookups. This isn't very good for concurrency especially since it's reliant on network traffic. If you had a few lookups in the queue, you could end up having goroutines waiting a few seconds or for lots of lookups this could balloon out to minutes for a lookup. This is the trade off. In general most programs aren't making thousands of concurrent DNS lookups.
Re:Network Programming with Go by dillonmacc on Monday June 11, 2012 @11:04PM (#4098484)
i just use http://www.unlocktheinbox.com/dnslookup/mx/ or some other online dns lookup tools.