News for hackers. Stuff that matters.
Re:Threads and Goroutines by nsm on Monday July 24, 2023 @08:18PM (#36856060)
I feel like this glossed over how when entering something like epoll, Go can optimize switching to another goroutine. Wouldn't the epoll block until the calling kernel thread had a file descriptor ready?
Re:Threads and Goroutines by sharkbot on Monday July 24, 2023 @08:41PM (#36856251)
I don’t have your answers, but the runtime source has some pointers that would likely get you the answer you seek: https://github.com/golang/go/blob/master/src/runtime/HACKING...

It sounds like blocking calls switch to a system stack and return the Go stack to the executor pool, but I don’t have source links to back up that claim.

Re:Threads and Goroutines by seabrookmx on Monday July 24, 2023 @09:12PM (#36856520)
The key is that epoll _isn't_ blocking: epoll_create returns immediately with a file descriptor. Go can use it to issue some sort of i/o operation and rather than blocking the kernel thread, it swaps it to another goroutine that needs work done. When epoll notifies the go scheduler the task is complete via that file descriptor, it will then resume the original goroutine.
Re:Threads and Goroutines by nsm on Monday July 24, 2023 @09:23PM (#36856628)
The crucial operation here is epoll_wait, not epoll_create.

Are you saying that the system call is performed on another kernel thread while goroutine continue running on the GOMAXPROCS threads?

Re:Threads and Goroutines by gpderetta on Tuesday July 25, 2023 @07:13AM (#36860622)
So, I'm not familiar with the go event loop specifically, but I have written a few event loops myself.

You can think of an event loop multiplexing across multiple sources of events: for example a local ready queue, a remote queue (for tasks coming from other threads), an io-queue, a timer queue. Epoll is used for the io-queue. If the local ready queue is empty (i.e. there are no runnable coroutines), the event loop will block in the epoll call, possibly until the next timer expiration if any; remote queue operations will wake-up the poll via somethink like event-fd. If the local queue is non-empty, epoll is called with a 0 timeout every N local queue pops (actual scheduling logic and priority is application specific of course).

Polling epoll with a 0 timeout is a bit wasteful, for high performance applications you can use io_uring so checking for non-empty queue is very cheap and/or use some network bypass library that makes epoll cheap. There is also old-school readiness notification via signals but I doubt that still see any use.

As you suggest, you could also have a dedicated thread per event loop just running poll (for example a sibling hyperthread), but it seems wasteful.

Re:Threads and Goroutines by muxator on Wednesday July 26, 2023 @12:14AM (#36873009)
Thanks for this comment! It is a very helpful design document for improving one's future event loops architecture!
Re:Threads and Goroutines by ridiculous_fish on Monday July 24, 2023 @09:37PM (#36856759)
Go can multiplex goroutines into a single syscall invocation. Two goroutines waiting on distinct file descriptors will use a single epoll (or similar) syscall. However not all syscalls can be multiplexed this way.

IMO this partially undermines the "Go doesn't have colored functions" claim. 1k goroutines calling read is fine, because read is multiplexed, but 1k goroutines calling stat will be painful, because you get 1k kernel threads. Go's "red functions" are those which require a dedicated kernel thread.

Re:Threads and Goroutines by insanitybit on Monday July 24, 2023 @11:47PM (#36857626)
I think it just undermines "colorless functions are good", not that go's functions are colored - they aren't. The problem is that "colors" (ie: "this is async") are helpful to the caller, they let them know which context they should be in when they perform that work.
Re:Threads and Goroutines by thumbuddy on Tuesday July 25, 2023 @06:45AM (#36860433)
I wish this was stated more. A lot of people new to go have this air of mystery surrounding what coroutines and goroutines are. They'll do something it'll be fast! Then do something else and feel like they are running hardware from the 90s. Not everyone has the know how to strace their applications and think through why something at that level is happening.
Re:Threads and Goroutines by gpderetta on Tuesday July 25, 2023 @06:57AM (#36860525)
Does Go spawn a thread for every non multiplexable system call?

In principle, on linux, you should be able to multiplex stat anyway via io_uring.

Re:Threads and Goroutines by wahern on Monday July 24, 2023 @11:18PM (#36857439)
epoll_wait takes a timeout, which if 0 means it will return immediately even if no events are pending.

You wouldn't necessarily need to make use of this feature to implement something like Go's scheduler. Jumping to the semantics of epoll skips most of the interesting bits about how it all works. The source code will always beat a blog post. If you're curious, you could work backwards from here: https://go.dev/src/runtime/netpoll_epoll.go Note that interface does indeed expose the timeout argument to the higher-level scheduling code; see line 92.

Re:Threads and Goroutines by nsm on Tuesday July 25, 2023 @01:10AM (#36858173)
I am aware of epoll_wait and friends (select/poll) accepting timeouts, and how that plugs into a general event loop like in libuv.

However if memory serves, these calls are reasonably expensive (on the order of microseconds, and _worse_ if readiness requires putting the current thread to sleep and then waking it up again). However https://www.ardanlabs.com/blog/2018/08/scheduling-in-go-part... says goroutines are closer to 200ns.

However the same source above does a decent job of explaining how epoll is being called "out of band". I am guessing when they talk about context switching costs, they are doing some sort of amortization of these system call costs. Since the actual OS threads try to remain CPU-affine (thread-per-core, LMAX disruptor etc. etc.), a lot of the OS scheduling costs can be skipped.

Thank you for the pointer to the source code. It gives me a starting point.