News for hackers. Stuff that matters.
Re:Go Concurrency Patterns ported to Clojure by _halgari on Monday September 16, 2013 @10:23AM (#6393387)
Using Thread/sleep inside of a go block is questionable. Using >!! and <!! inside a go block is simply wrong. Please switch to <! and >! and if you aren't simulating actual work, remove Thread/sleep. Failing to do so can seriously mess with the core.async thread pool.

Doing blocking operations inside a go is a anti-pattern.

Re:Go Concurrency Patterns ported to Clojure by pron on Monday September 16, 2013 @12:47PM (#6394201)
You could also use Pulsar (https://github.com/puniverse/pulsar). It has the same API as core.async, but none of these limitations (only you would need to call Fiber/sleep or Strand/sleep instead of Thread/sleep). Other than that, !< and !!< etc. are interchangeable in Pulsar (and either could appear in a function called by the go block; they don't need to appear in the same expression). You could then also do IO in go blocks.

But actually, Pulsar's main advantage is that it's simply a Clojure API for Quasar (https://github.com/puniverse/quasar), which adds all of these capabilities to Java and any other JVM language by providing true lightweight threads. core.async's advantage is that it can also be used in ClojureScript, while Pulsar only works on the JVM.

Re:Go Concurrency Patterns ported to Clojure by jbert on Monday September 16, 2013 @11:07AM (#6393608)
> Doing blocking operations inside a go is a anti-pattern.

Sorry - could you elaborate please?

In my understanding, in golang it's fine to block a goro with a blocking call, whether that be a blocking syscall (read/write/sleep) or a blocking read from/write to channel (perhaps multiplexed with select).

The scheduler will run other goros on the threadpool until the blocking call finishes. That's the "secret sauce" which allows you to write synchronous code in go without single-thread-per-goro (and requires a runtime to arbitrate the system calls and schedule goros)

Re:Go Concurrency Patterns ported to Clojure by masklinn on Monday September 16, 2013 @11:30AM (#6393728)
> In my understanding, in golang

The distinction is that it's not go, `go` blocks are macros doing a bunch of transformations. `<!` and `<!!` have the same purpose (get a value from a channel) but not the same context, `<!` can only be called inside a (go) block and will be transformed into a parking of the thread (yielding to the runner), whereas `<!!` works at a lower level and will block their whole thread.

> The scheduler will run other goros on the threadpool until the blocking call finishes.

`<!!` and the coroutine scheduler are not aware of one another, calling `<!!` from within a coroutine will block the scheduler. `<!!` (and `>!!`) exist specifically to be called from outside the scheduler (from outside a `go` block)

Likewise, `Thread/sleep` is not aware of the coroutine scheduler (it would have to be monkey-patched), calling it will also block the scheduler itself. core.async provides `timeout` for the purpose of parking a coroutine.

Re:Go Concurrency Patterns ported to Clojure by jbert on Monday September 16, 2013 @12:16PM (#6393984)
Thanks. I misunderstood the original comment to refer to golang code. The edited comment is clearer.
Re:Go Concurrency Patterns ported to Clojure by _halgari on Monday September 16, 2013 @11:25AM (#6393696)
In core.async code inside (go) blocks are run via a fixed thread pool (a large one, but still fixed), when you use <! and >! the code after the put/take is turned into a callback that is attached to the channel. The thread is then released back to the pool. This allows for extremely lightweight async code. In addition, goes can actually be GC'd when they deadlock and can never continue (all references to the channel a go is taking from are dropped).

Code inside (thread) blocks run on a dedicated thread. This allows for long running or IO bound processes to execute without clogging the main thread pool. Thread blocks do not rewrite their bodies into a state machine (unlike go blocks) and as such must use <!! and >!!. Since these threads are dedicated they do not return the thread to the thread pool when blocking. This allows for slightly better performance at the cost of an entire JVM thread for the life of the block.

TL;DR - Use <! and >! with go. Use <!! and >!! with thread. Mixing the two can have very unexpected results. In fact, using <! inside a thread block will result in a runtime exception.

Re:Go Concurrency Patterns ported to Clojure by bkirkbri on Monday September 16, 2013 @01:39PM (#6394519)
I've enjoyed working with core.async once I understood these constraints. It would be nice for code to "know" whether it's within a go block or not. That would alleviate needing do-something! and do-something!! versions of some functions.

By "know" I mean something like an in-go earmuff var.

EDITED for clarification and formatting

Re:Go Concurrency Patterns ported to Clojure by masklinn on Tuesday September 17, 2013 @03:35AM (#6398089)
> It would be nice for code to "know" whether it's within a go block or not.

Part of it obviously does since `<!` can't be called outside of a go block. And cljs has somewhat less of an issue there as it does not have `<!!` (since without extension the language has very little allowances for blocking)

Re:Go Concurrency Patterns ported to Clojure by jbert on Monday September 16, 2013 @12:15PM (#6393979)
Thanks. I misunderstood your (original, unedited) comment to refer to a golang goro (go func()), not clojure go macro.

Yes - this seems to be a key difference of golang's approach. The runtime+scheduler conspire to make a blocking goro much cheaper than an OS thread.

Re:Go Concurrency Patterns ported to Clojure by sanderjd on Monday September 16, 2013 @11:40AM (#6393777)
So, it seems to me that the goal of the original video was "look how easy it is to do various concurrency patterns in Go", and presumably the goal of working through Clojure versions of the same is "let's see if it's just as easy to do the same using Clojure's core.async!". In light of this discussion, I think that no, it is not as quite as easy (or simple?), and the subtle ways in which it is harder can be deceptive.
Re:Go Concurrency Patterns ported to Clojure by _halgari on Monday September 16, 2013 @12:17PM (#6393987)
Actually it's that you have more options in core.async. In some cases you want dedicated threads, other times you want lightweight threading. Core.async allows for both models.

Simplicity doesn't always mean "one" it means decoupling related ideas. Sometimes simpler means more things that each do one thing.

Notice how Clojure has refs, atoms, vars, and agents. Each does something completely different, each with its own tradeoffs and bonuses. Coupling reduces flexibility by throwing all the tradeoffs (often with additional issues) into one large pile.

So I see core.async as being inline with the Clojure ideals. Pull things apart and offer only simple primitives that can be combined to create larger systems.

Re:Go Concurrency Patterns ported to Clojure by icebraining on Monday September 16, 2013 @01:46PM (#6394570)
In golang you can just do LockOSThread() to reserve the OS thread for your current goroutine. What's the advantage of having two separate models and APIs?
Re:Go Concurrency Patterns ported to Clojure by pcwalton on Monday September 16, 2013 @01:42PM (#6394540)
I think this will practically always be true when porting a tutorial designed for language X to language Y, because tutorials designed with one language in mind tend to play to that language's strengths. Porting a Clojure-oriented tutorial to Go would demonstrate the same effect.
Re:Go Concurrency Patterns ported to Clojure by sanderjd on Monday September 16, 2013 @07:07PM (#6396488)
Very fair point.
Re:Go Concurrency Patterns ported to Clojure by xaritas on Monday September 16, 2013 @12:45PM (#6394188)
> Using Thread/sleep inside of a go block is questionable. Using >!! and <!! inside a go block is simply wrong.

They both block the thread and any other go-blocks that happen to be mapped to it. I think you get 2 x NUM_CPU threads in the dedicated pool for channels, so either of these is a mistake. (Aside: I'm not sure why you were down voted for saying this.)

When you need to wait inside of a go-block, the correct way is to use (timeout millis), which creates and then closes a channel.

This has some nice examples: https://github.com/clojure/core.async/blob/master/examples/w...

Re:Go Concurrency Patterns ported to Clojure by ryanbrush on Monday September 16, 2013 @01:20PM (#6394394)
Understanding that core.async is new and evolving, but is there a reference or an idiomatic way to work with blocking operations, beyond the general advice of separate threads as an exercise for the reader?

EDIT: I bring this up because this seems like a common misstep for users of core.async, and a simple pattern for blocking IO might help people avoid it.

core.async really looks awesome...if we can create simple patterns for working with blocking operations, it would be a compelling model for a lot of use cases.

(Also, hats off to the core.async developers; it seems like the reward for creating something great is just demand for more. ;)

Re:Go Concurrency Patterns ported to Clojure by bkirkbri on Monday September 16, 2013 @01:53PM (#6394623)
It's true that out of the box core.async is MUCH nicer to use with existing asynchronous IO APIs. My limited experience has shown that you'll want to wrap your synchronous IO operations to play well with core.async. This generally means queueing work in an operation-local (or subsystem-local) threadpool and providing an async API for interfacing with via core.async.

The next step is deciding on how to expose the asynchronicity. I chose to accept / return channels rather than callbacks, because... well... wrapping your sync code in a callback-based async API just to glue them together with go blocks and channels seemed backward to me. :)

I think this "next level up" from the foundation of core.async has potential for some really great solutions and I'm dying to see what Tim and others will come up with...

Re:Go Concurrency Patterns ported to Clojure by _halgari on Monday September 16, 2013 @01:36PM (#6394501)
I'm working on that right now, but it won't be out till the end of the year. I'm giving a talk at Clojure/Conj about core.async and hope to cover most of these sort of questions. I'd publish notes about it now, but that'd kind of kill my talk in two months.
Re:Go Concurrency Patterns ported to Clojure by ryanbrush on Monday September 16, 2013 @01:44PM (#6394551)
Sounds great. I'm glad to know something like that's in progress, and many thanks for your efforts on this project!
Re:Go Concurrency Patterns ported to Clojure by adambard on Monday September 16, 2013 @01:26PM (#6394434)
https://github.com/clojure/core.async/blob/master/examples/w... seems to be the only "documentation" I've found.