Why you can have millions of goroutines but only thousands of Java threads

Re:Why you can have millions of goroutines but only thousand... by toast0 on Tuesday October 09, 2018 @01:48AM (#18173610)

(disclaimer: I know very little about Go)

An expected way to manage this would be to have a limit on the number of threads to run blocking syscalls. Up to you how to do it (threadpool for syscalls, anythrrad can run a syscalls but checks a mutex first, etc)

I don't think there's a danger of deadlocks here -- your blocking syscalls shouldn't be dependant on other goroutines. Eventually the calls will succeed or timeout (one hopes) and the next call will commence.

In my experience, you can usually find a safe number of parallel syscalls to run -- and it's often not very many.

[ Parent | on HN ]

Re:Why you can have millions of goroutines but only thousand... by ridiculous_fish on Tuesday October 09, 2018 @03:02AM (#18173867)

Imagine 1000 pipes, each of which has a goroutine calling write() and another calling read(). If you schedule all of the writers and none of the readers (or vice versa) you'll get a deadlock.

[ Parent | on HN ]

Re:Why you can have millions of goroutines but only thousand... by leaveyou on Tuesday October 09, 2018 @03:51AM (#18174057)

AFAIK when a goroutine "calls write" (on a channel or on a socket, stream, mutex etc).. and the write "is blocked" it yields the execution and the scheduler can activate something else.. which can be a reader goroutine (after all the writers are blocked for example). So there's no deadlock as long as you have at least one reader for any writer.

[ Parent | on HN ]

Re:Why you can have millions of goroutines but only thousand... by gpderetta on Tuesday October 09, 2018 @05:13AM (#18174383)

That requires the underlying syscall to support an async or non blocking mode though. Disk io or stat doesn't on linux for example. The usual alternative is some sort of background pool for blocking tasks (which adds non trivial overhead), or, where supported by the OS, scheduler activation.

[ Parent | on HN ]

Re:Why you can have millions of goroutines but only thousand... by leaveyou on Tuesday October 09, 2018 @06:11AM (#18174611)

You are right, see Synchronous System Calls in

https://www.ardanlabs.com/blog/2018/08/scheduling-in-go-part...

[ Parent | on HN ]

Re:Why you can have millions of goroutines but only thousand... by toast0 on Tuesday October 09, 2018 @09:17AM (#18175671)

Sorry, I also expected those things that can be translated into non-blocking calls with select-ish apis for readiness to be translated as well -- because it's the right thing to do in this type of environment.

To be more specific, most socket calls should be async + selectish, but file system calls would likely be done as synchronous, because async doesn't generally work for that -- anyway limiting the number of outstanding file I/O requests tends to help throughput.

[ Parent | on HN ]

Re:Why you can have millions of goroutines but only thousand... by zbentley on Tuesday October 09, 2018 @02:26PM (#18178520)

stat() cannot (on platforms I am familiar with) be performed in a selectish/nonblocking way. That call can block forever. Local filesystem access in general is still not async-capable in very important ways.

[ Parent | on HN ]

Re:Why you can have millions of goroutines but only thousand... by toast0 on Wednesday October 10, 2018 @02:00AM (#18182365)

Yes, filesystem calls are going to block -- but putting that into a limited threadpool shouldn't deadlock your system -- you can't do a blocking read on a file, waiting for it to be filled by another thread (unless/until you start playing with FUSE, I guess.

[ Parent | on HN ]