Sounds like you've basically explained several of the common concurrency features of Haskell. Haskell has the advantage that any pure data structure (ie, not using mutable references, which is almost all data structures we use) can be used concurrently and atomically using IORefs and atomicModifyIORef. If you need transactional guarantees, then software transactional memory (STM) is just as easy to use. For both these situations, it's laziness and purity that gives us these things essentially for free.

Of course you could create more advanced concurrent mutable structures as other languages have, but IORefs and STM get you a very long way when coupled with GHC's extremely light weight threading and high performance IO (which is getting even better in GHC 7.8, to be released soon).

To learn more, Simon Marlow's excellent book is available for free online [1] and in dead tree and eBook forms from O'Reilly. It's very clear, with a great balance of how to use things practically, how they work, and how to make them work their best.

[1] http://chimera.labs.oreilly.com/books/1230000000929/index.ht...

There are also things like acid-state [0] if you want transactional, in-memory, and persisted data stores.

[0] http://hackage.haskell.org/package/acid-state

Speaking of the JVM, Scala has immutable data-structures. You stick one of those suckers into an AtomicReference and presto you've got a thread-safe data-structure. And for more complicated scenarios, you also have Scala-STM: http://nbronson.github.io/scala-stm/

Even though the concurrent data-structures in Java's library are efficient and well built, they remain error-prone because thread-safe operations are not composable. Working with immutable data-structures in a high concurrency scenario is the best thing since sliced bread.

I do agree with your point - sane concurrency implies using the right abstractions for the job and there are cases in which you need shared memory storage. The JVM is in fact great because it does 1:1 multithreading, because the GC is so good that it allows you to be a little wasteful and because it provides the required concurrency primitives to build whatever abstraction on top that you desire.

For Scala + the JVM, just to enumerate - you've got actors by means of Akka and it's pretty capable as it's flexible and it allows you to manipulate/coordinate entire clusters of remote actors (and of course there's Quasar that does the lightweight threads thing and I'd love a comparison between the two btw), you've got the Future/Promise concept baked in and with a sane design [2], with C#'s "async" by means of a library that will be part of the standard [3], you've got great immutable data-structures that also have parallel extensions [4], you've got RxJava/RxScala for processing streams of events reactively [5], or Iteratee IO for safe processing of data streams [6], or shared transactional memory that I mentioned above [7] and of course things inherited from Java, like NIO which in spite of its flaws it's the sanest cross-platform abstraction over native APIs for async I/O, plus as I've said, the best cross-platform concurrency primitives you can get (e.g. I still get boners every time I use ReentrantReadWriteLock).

[1] http://akka.io/

[2] http://docs.scala-lang.org/overviews/core/futures.html

[3] https://github.com/scala/async

[4] http://docs.scala-lang.org/overviews/parallel-collections/ov...

[5] https://github.com/Netflix/RxJava

[6] http://www.playframework.com/documentation/2.2.x/Iteratees

[7] http://nbronson.github.io/scala-stm/

> ... immutable data-structures. You stick one of those suckers into an AtomicReference and presto you've got a thread-safe data-structure.

Thread-safe – yes (and Clojure does this beautifully; in fact any Clojure DS is thread-safe); concurrent – not really, because you can only have one writer at a time, which simply doesn't scale for large data structures.

> I'd love a comparison between the two btw

Quasar has true lightweight threads (M:N multithreading as in Go or Erlang), and an (optional) actor system built on top. Quasar actors can and should block as much as you want, have selective receive (just like Erlang), and are especially easy to use. Also, Quasar has Java and Clojure APIs that feel natural and idiomatic in both languages.

Erlang has ets, which meets some of your requirements (not transactional, but mnesia is)

And Rust has the `sync` crate which has a variety of concurrency mechanisms and primitives: http://static.rust-lang.org/doc/master/sync/index.html

No STM of course, but RWArc allows for multiple writers, or you can just uses Arc if immutable sharing is acceptable, or just message passing via Chan (in the standard library) if sharing data concurrently is not required. Lots of options.

But those are just what they are - primitives, rather than data structures. RWArc allows just a single writer.

You are correct, you'd use them to implement your own concurrent data structures per requirements. But I do think they're slightly higher level primitives than most languages provide, and there is some nice flexibility in the options available, in terms of control over sharing/mutability and mechanism.

RWArc allows multiple writers, but each blocks until the "active" writer gives up its lock. But I suppose you mean that you can't have concurrent writes, so I understand this might not be suitable for the use-cases you were talking about.

I think the Rust people have made the right choice at every juncture given the language requirements. It's a very impressive language, but it, too, has its sweet spots. There's a reason why Java has attracted so much concurrency research in the past decade, and why it's usually years ahead of almost any other platform in that respect: no, it's not (just) the platform-agnostic memory model, which later inspired C++11's (heck, it was done by the same people), but its world-class garbage collector(s). While it's possible to implement lock-free data structures without a good GC (with RCU), it's very difficult to do so in a general enough manner while still staying efficient. Java's GC, along with its memory model and direct access to CPU concurrency primitives (CASes since Java 5 and fences since Java 8), combined with Doug Lea's inspired implementations[1], have made the JVM the premier choice for anyone who wants to take advantage of cutting-edge concurrency.

[1]: http://www.youtube.com/watch?v=sq0MX3fHkro

EDIT: The Doug Lea talk linked above is probably the best overview of what doing low-level concurrency on modern hardware entails. In particular, it demonstrates interactions among abstractions layers and dispels myths that claim that having less software layers is always better (hardware already has so many layers as it is, that some software layers can actually help, if only to hide the hardware's complexity).

Thanks for the link to the Doug Lea screencast/talk, very interesting so far.

EDIT: Indeed it was a great talk, very informative! I came to the opposite conclusion about layers of the system, though I'd say that it isn't the number of layers that's the problem, but the nature of them.

From the talk, having the GC insert safepoints where threads can be stopped for a collection, and having the JIT inserting counters (resulting in memory contention) are two things in Java that caused for high-performance concurrency. Running on top of a VM causes similar issues. Though, clearly there are many other layers and unanticipated layer interactions, just laying in wait to mess you up regardless.

Well, he only mentions how these things make his life difficult, not how they make it easier. When you consider something, you must also consider the alternatives: a GC does insert safepoints on the one hand, but it makes lock-free data structures possible (or much, much easier) on the other. The JIT counters introduce contention before optimization kicks in, but then you get virtual method inlining, the mother of all optimizations, which you can't do without a JIT, certainly not when you want to support hot code loading.

Those things are necessary for the class of programs Java SE is meant to support (long running, high-performance server-side applications); they're not ideal for device drivers, command line programs or fast-loading GUI apps, which Rust is designed to handle. Everything is a tradeoff.

No, RWArc definitely allows more than one writer. One writer at a time, sure, but you can have multiple tasks try to write to it at the same time and the first will get access while the others will block.

EDIT: Oh, I see what you mean now; sorry, I misinterpreted. Carry on.

I remember reading some criticism about the STM: https://groups.google.com/forum/#!topic/clojure/XHqWLMcsH-c

I wonder if nowadays people can generally use it without never having to think about locks?

Isn't this the primary use case for Clojure's agents?

No. Like I said, agents allow a single writer at a time. Refs are more like it, but it's really hard to build a good concurrent data structure (like a B tree) efficiently with refs.

As far as I understand, refs are not intended for implementing data structures. Instead, they should be used to provide concurrent access to immutable data structures. So the "Clojure way" solution would be to efficiently implement an immutable B-tree and then expose a ref to it.

An atom would do that much more efficiently that a ref. Refs are there for more complex types of mutation (like mutating more than one persistent DS transactionally).

Also, B-trees are a mutable data structure. "Immutable" B-trees aren't B-trees. Clojure's own persistent DS tries are quite efficient, but not nearly as scalable as a concurrent B-Tree, which easily supports lots of concurrent writes. AFAIK, there are no persistent data-structures that efficiently scale writes.

Well stated.