News for hackers. Stuff that matters.
Re:Clojure Concurrency Tutorial by tombert on Tuesday April 09, 2019 @01:07PM (#19616765)
To a reasonable extent, yes, they're the same. The biggest issue is that goroutines are (more or less) fully preemptive (for all intents and purpose) with their thread pooling and go blocks in core.async are not. Basically, in Go, you can have as many blocking IO thing taking millions of seconds to run, but it won't block the thread pool, and your other goroutines will run fine.

Since core.async go blocks aren't fully preemptive, long-running things can eat up the whole thread pool. core.async allocates "number of cores + 2" number of "real" threads to do its work.

For example, say I have a program that downloads hundreds of giant files running on an 8-core server. The naive implementation would have a goroutine/go block for each concurrent download. In Go, this would work more or less ok (assuming you have plenty of memory), but in Clojure it would block all the other go blocks after 10 until the download is done.

Normally, it's not an issue, but it's a small gotcha if you're not paying enough attention.

Re:Clojure Concurrency Tutorial by jwr on Wednesday April 10, 2019 @01:10AM (#19622036)
This is true, but I think it's worth mentioning that core.async lets you freely mix "full threads" with work scheduled from a thread pool. You use the same channels for both. You do the lightweight stuff in go blocks, and the I/O heavy work in a `(thread)`, where you use `<!!` and `>!!` for blocking gets and puts, respectively.

I find this to be amazingly flexible. You are right in saying that it's a gotcha: you do not want to do I/O heavy work in go blocks in general.

It is also worth noting that core.async works just as well in ClojureScript, which when you thing of it is pretty amazing. No threads there, of course.

Re:Clojure Concurrency Tutorial by quadcore on Tuesday April 09, 2019 @01:21PM (#19616939)
I'm a bit surprised if I understand you correctly. I though a goroutine would run and hold the thread until any kind of sleep.

edit: Oh but there might be no "core.async sleep" in most of clojure libraries like IO/net stuff, which would explain why async go blocks hold the thread more often than they should, whereas Go has "goroutines sleeps" everywhere.

Re:Clojure Concurrency Tutorial by setr on Wednesday April 10, 2019 @02:45AM (#19622411)
As I recall it, goroutines are in fact cooperative with implicit yield points. The implicit behavior is what I believe go means by “for all intents and purposes”

Particularly, I believe all function calls are implicitly yield, which means you can usually pretend that its preemptive through normal coding practices; but you can still never yield through a while(1){} (i think there was a proposal to add implicit yield on loops? Not sure if it went through)

I would assume core.async does not have many, if any, implicit yields, leading to the different behavior. But technically, both are cooperative, go is just more convenient about it

Re:Clojure Concurrency Tutorial by quadcore on Wednesday April 10, 2019 @01:49PM (#19627091)
The implicit yield points you're talking about is sleep or wait (noop in other terms). It's also not exactly a yield, it calls the scheduler to tell it to schedule (meaning the scheduler could decide to run again the exact same goroutine that noop-ed). Finally, goroutines are not just cooperative, they also run in parallel in a pool of threads.
Re:Clojure Concurrency Tutorial by setr on Wednesday April 10, 2019 @04:17PM (#19628379)
>The implicit yield points you're talking about is sleep or wait (noop in other terms)

Isn’t sleep() usually defined as not giving up control? It’s semantically a busyspin for n seconds. That is, it’s explicitly not a yield point; it's a blocking call, and it doesn’t give control back to scheduler.

And isn't wait() usually defined as, semantically, a non-blocking spinloop with a condition (hence notify() or poll())?

Which makes sleep(), wait(), yield() and noops very different.

I suppose in go, if yield() is implicit on any function call, then sleep() will immediately call yield() before actually sleeping (but once started, it’ll hold control until finished)

I'm pretty sure the implicit yield points are actually yield(), though I haven't verified; it's the only one of the sleep/wait/yield semantics that would be sensible to be implicit as far as I can tell.

>It's also not exactly a yield, it calls the scheduler to tell it to schedule

As far as I was aware, thats exactly what yield means; it yields execution control back to the scheduler. I suppose in eg, a generator, it yields control back to the caller, but afaik thats fundamentally the same meaning (the caller is the scheduler). Can you expand?

>Finally, goroutines are not just cooperative, they also run in parallel in a pool of threads.

Sure, but is that relevant? I assume async.core also trivially supports thread pooling, and is presumably equivalent to go’s pool. The only real difference is go implicitly instiates the pool, while core.async presumably makes it an explicit function to call

Re:Clojure Concurrency Tutorial by quadcore on Wednesday April 10, 2019 @11:55PM (#19631037)
In Go, time.Sleep don't hold the thread. That's the cool part. In Go a noop gives the control back to the scheduler which will then reuse the thread for any goroutine that needs cpu -- ie. one that isnt noop-ing. So if you sleep or wait for 10 seconds, 10 seconds of cpu will be given to other goroutines.
Re:Clojure Concurrency Tutorial by Blackthorn on Wednesday April 10, 2019 @09:51AM (#19624579)
The real magic is where you can call blocking functions from goroutines and have it Just Work!
Re:Clojure Concurrency Tutorial by chongli on Tuesday April 09, 2019 @11:58PM (#19621710)
What's the advantage of downloading multiple massive files in parallel, assuming you can saturate the link with just one? Isn't it better to finish one file before moving on to the next?

If you can't saturate the link with one file, then you should consider something like bit torrent.

Re:Clojure Concurrency Tutorial by tombert on Wednesday April 10, 2019 @10:56AM (#19625306)
I'm just saying that as a dumb example of something that isn't terribly CPU intensive, but would block up the threads, but even then, let's assume that the downloads are really slow, so that it's not anywhere near saturating my network speed. The go-blocks in core.async would still block the other go blocks. This wouldn't be an issue with Go.
Re:Clojure Concurrency Tutorial by quadcore on Wednesday April 10, 2019 @01:52PM (#19627120)
I/O is exactly the kind of stuff that should happen smoothly in goroutines or go blocks. It's kind of the quintessential case because I/O has a lot of sleeps and waits (noop or stalls in other terms).

When you're downloading a big file for example, I'm not an expert of that but I expect that there's gonna be a lot of cpu stalls in there. Same for the network adaptor and disk.

Re:Clojure Concurrency Tutorial by kitd on Wednesday April 10, 2019 @04:33AM (#19622848)
It's an example. Think of something else, eg an HTTP client call that sits waiting for a response, instead. The call parallelism is the issue being discussed.