News for hackers. Stuff that matters.
Re:Proposal: Non-cooperative goroutine preemption by mseepgood on Thursday March 29, 2018 @09:10AM (#16706656)
> So, preemptive green-threads? This seems like a counter-productive and highly complicated re-implementation of what the OS thread model does, but inside the Go runtime.

You can't easily spawn 10.000 OS threads.

> I would very much like for goroutines to remain cooperative. It is much simpler, naturally more performant, and quite easy to reason about.

Cooperative is less simple to reason about, as the proposal explains. Preemption fully delivers what the developer already expects of goroutines.

Re:Proposal: Non-cooperative goroutine preemption by arghwhat on Thursday March 29, 2018 @09:35AM (#16706871)
> You can't easily spawn 10.000 OS threads.

Of course you can, without any problem at all. It's just more expensive than a green thread (they're full processes with stacks), and unlike green threads, it affects how processing time is sliced. It is, however, much cheaper than people tend to believe.

> Cooperative is less simple to reason about, as the proposal explains.

Cooperative scheduling is very simple to reason about. In the simplest form, you do not give up control unless you use I/O, a synchronization primitive, or explicitly give up control (runtime.Gosched() in Go).

As a Go developer, and as a developer in any other language, this is exactly what I expect from any green thread implementation. That's how M:N scheduling has always been implemented, and how green threads have been implemented in other languages.

Technically, preemptive is harder to reason about, but you save having to reason about it. However, if you expect preemption from goroutines, you have misunderstood the primitive.

Re:Proposal: Non-cooperative goroutine preemption by ngrilly on Thursday March 29, 2018 @10:56AM (#16707558)
> In the simplest form, you do not give up control unless you use I/O, a synchronization primitive, or explicitly give up control (runtime.Gosched() in Go).

This is not how Go works nowadays. There are other preemption points.

Re:Proposal: Non-cooperative goroutine preemption by arghwhat on Thursday March 29, 2018 @11:21AM (#16707758)
That's why I said "in the simplest form".
Re:Proposal: Non-cooperative goroutine preemption by ngrilly on Thursday March 29, 2018 @11:32AM (#16707861)
Yes, but it completely ruins the argument about "it's easy to reason about" ;-)
Re:Proposal: Non-cooperative goroutine preemption by akvadrako on Thursday March 29, 2018 @09:33AM (#16706859)
> You can't easily spawn 10.000 OS threads.

This is what I thought until last week, but it seems it's no big deal. 1M threads starts to be an issue.

Re:Proposal: Non-cooperative goroutine preemption by pcwalton on Thursday March 29, 2018 @04:21PM (#16710691)
> You can't easily spawn 10.000 OS threads.

Of course you can. Go on, compile and run this program:

    #include <pthread.h>
    #include <stdlib.h>

    void *f(void *arg) {
        pthread_mutex_lock((pthread_mutex_t *)arg);
        pthread_mutex_unlock((pthread_mutex_t *)arg);
        return NULL;
    }

    int main() {
        pthread_t threads[10000];
        pthread_mutex_t mutexes[10000];
        for (unsigned i = 0; i < 10000; i++) {
            pthread_mutex_init(&mutexes[i], NULL);
            pthread_mutex_lock(&mutexes[i]);
            pthread_create(&threads[i], NULL, f, &mutexes[i]);
        }
        for (unsigned i = 0; i < 10000; i++)
            pthread_mutex_unlock(&mutexes[i]);
        for (unsigned i = 0; i < 10000; i++)
            pthread_join(threads[i], NULL);
        return 0;
    }
Executes in a second for me, even on macOS.
Re:Proposal: Non-cooperative goroutine preemption by vidarh on Thursday March 29, 2018 @09:31AM (#16706847)
> You can't easily spawn 10.000 OS threads.

I just spawned 10.000 OS threads running separate mRuby instances in each thread on my laptop, to see if it'd cause any problems (because I happened to have an app sitting around that spawns separate threads; note: these are not Ruby threads, mRuby does not have built in threading). None at all. So yes, you can. I didn't check how much memory it'd take, however, or attempt to measure overheads in any way, so I'm not saying there aren't potential issues with it.

Re:Proposal: Non-cooperative goroutine preemption by jashmatthews on Thursday March 29, 2018 @09:39AM (#16706921)
Simply using 1,000+ OS threads is often very close to as efficient as using coroutines or an event loop: https://www.slideshare.net/e456/tyma-paulmultithreaded1
Re:Proposal: Non-cooperative goroutine preemption by zzzcpan on Thursday March 29, 2018 @09:46AM (#16706974)
If your event loop is as slow as 1000+ coroutines or OS threads, you are doing it very very wrong.
Re:Proposal: Non-cooperative goroutine preemption by jashmatthews on Thursday March 29, 2018 @09:52AM (#16707026)
AFAIK the main benefit to coroutines is getting to manually manage tiny stacks.

For example, in Go, they mark a goroutine's stack as "hasn't executed since the last time we marked", which makes GC with thousands of threads a lot faster.

Re:Proposal: Non-cooperative goroutine preemption by arghwhat on Thursday March 29, 2018 @09:54AM (#16707042)
[Citation needed]
Re:Proposal: Non-cooperative goroutine preemption by jashmatthews on Thursday March 29, 2018 @10:35AM (#16707370)
https://github.com/golang/go/issues/12061

"Mark termination does not have to rescan stacks that haven’t executed since the last scan"

With pre-emptive scheduling of OS threads, you potentially dirty a ton more stacks, and have to scan them all for pointers to heap memory. I'm not sure how VMs using lots of OS threads deal with this.

Re:Proposal: Non-cooperative goroutine preemption by coldtea on Thursday March 29, 2018 @10:42AM (#16707433)
>So yes, you can. I didn't check how much memory it'd take, however, or attempt to measure overheads in any way, so I'm not saying there aren't potential issues with it.

The "easily" part wasn't referring to feasibility, but to how effortless it is to do AND have performance and scalable code.