Besides your listed reasons mutable state is becoming a bigger and bigger bottle neck as the number of cores goes up and that's why functional programming is becoming more and more important. It also makes programs much harder to write in a bug-free manner and it makes testing much more difficult (if not impossible).
It's one of the core reasons why I felt so bad about dropping Erlang from the list because I feel that it gives me that plus a battle tested environment to work with.
Still tempted to put it back in. Your reasons add weight to an already very difficult decision, whether or not to pursue this opportunity as a way to get more into functional programming or to stay in my 'comfort zone'.
I've been toying around with FP on and off for the last couple of years and it is one of the most important revivals the computing world has seen to date (imo of course).
Finally, I must say that while Scala does indeed have a first-class web framework in Play, web frameworks are first-class objects in Clojure. Joke. Sort of. I personally do not want to have a monolithic pre-packaged web framework to bow down to; the Clojure community is a community of hackers, we like to make things out of other things, and to take things apart. Clojure and the work of its community allows me to compose my own web framework as needed very very quickly–and to drastically change it just as fast. I'm not saying you should stop doing what you like to do, but if you gave Clojure a chance...dude: it's addictive.
The code I write now is nothing like Java, but it's a question of how you get from there to here. When I started Scala I wrote what I'd write in Java. In the first week I removed the semicolons. In the second week I started to use function literals. Then I started using match/case. Later I found out about for/yield with futures. And so on. I learnt the language at my job, and I was delivering real, user-facing features in it from day 1. I don't think I could've done the same with Clojure.
(Reasonable people can disagree on syntax; I find that Scala's lets you get very close to the language of the domain (e.g. see the Spray "routing DSL") and that's worth the cost it imposes on the macro system)
Also, sometimes you want your updates to a data structure to be seen via an alias (happens often actually), then we are back at square one again.
Systems people are much more pragmatic about this: they see the trends in hardware before we PL people do, and their solution has never been functional programming. Ok, well...maybe once with MapReduce (but Jeff and Sanjay are both systems PL people).
Adopting it successfully does require separating essential from accidental state which naturally leads to some variant of FRP being adopted to manage (re-)derivation of accidental state; which is usually the hard to explain part.
I am wondering what you do see systems people doing by the way? I'm always interested in broadening my perspective on this.
The systems people I work with do a lot of key value stores (sharded NoSQL-style databases), there is a lot of pipelining for data parallel processing (MapReduce, Pig), and there is still a lot of classic locking going on, throughput is important. Oh, and lots of Paxos, logging, state replication, etc... The normal stuff you could find at Google (note I don't work for Google).
On the HPC side, the people that really care about performance, there is some map reduce via MPI, but most of the big gains are to be had in CUDA, which is an extremely imperative programming model with very synchronized core executions and memory accesses.
I've learned that whenever the FP people talk about high performance, they usually mean something else (like concurrency that isn't so slow), rather than the real high performance stuff that the big kids (systems and HPC communities) are doing.
The reality though is that well over 99% of the market works with data sets for which, with current hardware, the performance costs of these persistent data structures are very much irrelevant.
When you add FRP on top of that so that you can keep your managed state minimal you can actually often achieve quite dramatic performance improvements over the traditional hand-written data-to-UI synchronisation.
For persistent data structures to come into their own for big data quite a few more things are needed of course; but if you look at Nathan Marz's Lambda Architecture for example it's little more than a handwritten system for making large scale persistent data structures manageable with today's systems already.
I'm fairly sure that tooling for that will only improve over time until the very thought of manually managing derivation pipelines becomes as quaint as manual memory management.
As for HPC; yes... I would not be surprised at all if at that level mutable state no longer is a premature optimization. Even there I could imagine many of the more high-level control structures benefitting from persistent data structures though.
About the FRP part; which competing paradigm are you working on? I wouldn't be able to name a 'competitor' so I'm quite curious.
Yes; the managed time approach in Glitch is very interesting and indeed a competitor towards addressing the same problem I originally raised.
The core of my original statement/problem definition was that real mutable state, i.e. throwing away history by default, was no longer the smart thing to do given current hardware.
From what I can tell you would also agree with that...
Why is it hard to get across in writing?
a) it requires putting together a fairly large number of concepts of moderate complexity (and you never know which ones your readers already understand.)
b) many of the concepts sound trivial and people usually don't grasp their power until you've listed X amounts of things it would enable.
And to be clear; it's hard for me to put in writing for now ;)
I would say that adopting a language for the next 10 years that doesn't have purely functional data structures at its core is a fool's errand.
Persistent data structures, i.e. efficiently storing every version of your data structure, is the natural reflection of the new reality in computing where:
Whether you adopt this through Haskell, Clojure, scala or even Bloom e.a. is fairly inconsequential.What will matter is that persistent data structures will inevitable become commonplace over the next decade and that techniques for leveraging it (such as functional reactive programming) will be adopted in one form or another.
@jacquesm: I live in Amsterdam so I can try to explain it to you in person once; virtually everyone still underestimates the impact this will have and (or perhaps because) it's very difficult to get that across in writing.