1. It's just plain easier to write a lot of stuff with a shared memory approach.
2. There are algorithms where you get the best speedup using shared memory.
But yes, there are also downsides.
1. Shared memory concurrency can be a lot harder on the runtime. In particular, writing an efficient single-threaded GC is an order of magnitude easier than an efficient concurrent GC.
2. You eventually run into scaling issues with shared memory only (number of cores, limited memory bandwidth). That said, a hybrid shared memory/message passing solution can still be superior to a pure message passing one, and for "desktop" parallelism that's not an issue.
3. Shared memory concurrency requires programming language support (at least if you want to keep a modicum of sanity), whereas message passing concurrency can be done as a library. And many languages have really, really screwed up their handling of shared memory concurrency where it's extremely difficult to reason about correctness.
My perceptions may be colored here because I did cut my teeth learning about parallel programming with distributed computing in the 1990s, but I also find that lack of shared memory concurrency less of an issue in actual practice (though, obviously, I'd hardly pass up on having the option).
I am puzzled why other languages do not even try to explore this. This will be a nice fit for Ocaml or nodejs. And even with Go I wish for be an option to run several Go processes with a shared heap with explicit API to write/read things there and channels working across processes.
The ability to just throw goroutines at the scheduler changes how you approach writing programs. For example, say you want to process an input that consists of lots of individual records. You can run the input stream in one goroutine, then spawn a bunch of worker goroutines that each receives records on a channel for processing. Since Go uses real threads and doesn't have a global lock, you can get nearly linear improvement in throughout here.
You wouldn't do this in, say, Ruby or Python, where the concurrency situation is about the same as in OCaml. Ruby has threads, like OCaml, but their slowness means they're not really usable in the same way as goroutines.
Parallelizing apps by forking child processes and communicating input and results via pipes is something I've done a lot in Ruby, and it's a really awkward, heavy-handed concurrency model.
Go really is designed around message-passing as its primary means of coordination; the key part here is having nice abstractions, which is where Go's strengths lie and which revolve primarily around `select`. But writing higher-level abstractions over processes and FIFO channels isn't terribly hard, as long as you have serialization built in (which Python, Ruby, and OCaml all do).
This is not true. Prior to Go 1.5 GOMAXPROCS defaulted to 1, but could always be set to use any number of cores. In Go 1.5, the default value for GOMAXPROCS was changed to the number of "cores" available on the machine. Prior to then many applications were setting GOMAXPROCS based on CLI flags or just hard-coding the future behavior with `runtime.GOMAXPROCS(runtime.NumCPU())`.
How does Lwt interact with third-party libs that aren't written to use Lwt? I know from Ruby that libevent-type stuff such as Eventmachine doesn't combine all that well with non-async code.
Go made a good call in making all I/O blocking. That means code is easy to read and write. It comes with downsides, such as that you have to use goroutines, but for most developers it's a more productive concurrency model. (Unfortunately there's no alternative; you can build a blocking system out of a non-blocking one with no overhead, but the opposite is not true. When I first started with Go I was surprised that you can't "select{}" on a file descriptor or socket, or indeed any custom implementation, in the same way that "range" is magical and only works on built-in types.)
Lwt does have proper cancellation support. There are some pain points, but it is straightforward to work around them. The real problem with Ocaml is that there are 2 incompatible libraries for promise-based IO, Lwt and Async. That made it really hard for third-party libraries as now they need to support not one, but two frameworks...
[0]: http://kcsrk.info/ocaml/multicore/2015/05/20/effects-multico...
[1]: http://kcsrk.info/ocaml/multicore/effects/2015/05/27/more-ef...
[2]: https://github.com/kayceesrk/effects-examples
[3]: See recent pubs here http://kcsrk.info/.
For typical server tasks, like web serving, we can just run several processes. Unix is doing this for decades. But even web serving is often cpu bound, if your SQL query runs only on one cpu.
The obvious thing to make programs faster is, to utilize as many cpus as possible to perform the computations. There is no silver bullet, but the recent years brought quite some progress. Go makes it easy and cheap to spawn off goroutines, the channels and GC help to make this a reasonable concept. Functional programming languages are even a more obvious candidate for parallel execution, as they have no side effects.
It actually feels like the recent years have been pretty disappointing. Go is one of the few places where the language developers are making a real effort around multicore. Erlang came with a bunch of opinions about how to do this in the 90s.
But every other mainstream language is taking a head-in-the-sand attitude about this. Python squandered their backwards compatibility break without addressing this at all. The JVM developers are making no effort to get away from the global heap. Ruby and Node are completely ignoring it entirely. PHP from it's request/response web history actually does well, except its non-web story is as dismal as ever. Lua has coroutines, which are a reasonable abstraction, but no multicore VM story...
But, these days using unix sockets with some cute flags letting multiple proceeds binding to the same listening sockets means it's sort of less needed!
Though, doing a lot of work with Nim lately and exposing thread pools via Futures makes for some pretty lovely easy parallel code, so I dunno. Not a deal breaker but a nice to have, and having threads in OCaml will make things just that bit nicer
Here's a concrete example: comparing the Isabelle proof assistant, written in PolyML (a multi-core supporting SML implementation developed mostly by one guy) with Coq (which is written in OCaml). Interactive theorem proving in the former is a lot nicer as it takes advantage of multithreading not only for faster concurrent processing of proofs, but also to do things like running Quickcheck and Nitpick in a separate thread automatically to identify trivially falsifiable lemmas, and automatically finding stdlib lemmas that exactly solve a particular proof.
I think PolyML demonstrates that a lack of manpower isn't what stopped OCaml implementing multithreading support. Over the years there have actually been a few proposals/branches implementing some form of support, but all were rejected/abandoned. There was even one that just made the runtime reentrant (passing the runtime around directly instead of having it as a global variable), meaning the OCaml runtime could now be stopped and restarted when embedded in another application (e.g. C calling OCaml), a feature already present in Haskell, but this relatively simple improvement was also rejected (which personally bothers me a lot on a subjective level as I hate globals so it seems like a worthy improvement for its own sake). I remember waiting excitedly four or five years ago for it to be merged, a small but significant step on the path to multicore support, only for that hope to fade away as contributions to the branch slowed to a trickle then dried up completely. Without such extreme focus on avoiding decreases in single-core performance, it would have been much easier for a change like this to have been merged.
I think in recent times there is even less justification for rejecting multicore for affecting single-threaded performance. For two reasons: firstly, the recent addition of FLambda has brought performance improvements in many cases of over 10%, easily enough to compensate for any loss from multi-threading. Secondly, HFT now basically requires FPGA to compete, so single-threaded performance would presumably be of less concern to Jane Street (OCaml's biggest industrial user) now as they really shouldn't be using a software execution engine anyway (disclaimer: I say that as someone working at a competing HFT firm). And if Jane Street isn't doing HFT, then a drop of a few percentage points in single-threaded performance shouldn't affect them much anyway.
Finally, OCaml could always do what Haskell did and add a flag to toggle whether or not multi-threading is enabled, allowing single-threaded users to avoid any performance regression.
In my experience with Isabelle, these tools provide feedback on the order of seconds (because they do expensive things). Maybe several hundreds of milliseconds, for simple cases. At these scales, it makes no noticeable difference what kind of parallelism solution you use. Spawning separate processes and sending messages would work just as well.
So the reason there is no parallel Quickcheck for Coq is more likely due to the fact that there is no Quickcheck for Coq, period.
In recent years where servers are to scale to many machines I found myself using less and less kennel threads. Also it seams GUIs are mostly web theses days. Most remaining fields I can imagine where you would want kennel threads, ocaml would not fit anyway because of GC and lack of control over memory layout.
So I really wonder...