I have news for you. Redis and Node.js are widely used because they are able to solve real problems that real people have. They work in practice despite your personal opinions to the contrary.
Moving on, you're dead wrong about concurrency. How many programs are running at once on your computer? While you are loading a web page, do you want your browser to continue to be able to respond to you? Those are two simple cases where people want concurrency. Those are also two cases that Celery doesn't handle. At one point they would have been written with cooperative multi-tasking. Today these are generally written with preemptive instead. (Though there are some exceptions. PuTTY is a fun one. That implemented coroutines in C with an impressive preprocessor hack. See http://www.chiark.greenend.org.uk/~sgtatham/coroutines.html for details.)
And back to Celery, things that don't require locking, can and should get by without locking. This is a useful observation, but hardly a universally applicable one. And waving the magic Celery want doesn't change that. If I have 500 files to process, and that data needs to wind up nicely summarized in a database, Celery will only let me get rid of locking in my code because I rely on locking being implemented in the database.
If you care about data and race conditions, at some point you have to solve hard problems. And you'll need to lock. But only do it when you need to.
Opinions of mine were made after trying both Redis and Node.js and finding them in poor state. You can read about Redis's problems here: http://redis.io/topics/latency and here http://redis.io/topics/transactions
Node.js is better in regard to handling high load. And the concept of having the heavy lifting happening mostly away from programmer is very sensible. I only find the implementation lacking and there seem to be a fair amount of people now migrating away from Node.js
In regard to your point about two programs running "concurrently" - that's actually multitasking, a feature of OS, not the programming language. And if your OS doesn't support that, you can't write a web browser that will make the email client work concurrently.
Concurrency only means that two different things seem to be happening at once.
Cooperative multitasking and cooperative concurrency are two different things. Cooperative multitasking means that if your browser is poorly written and blocks - your whole machine blocks.
On the other hand cooperative concurrency is a paradigm under which if your coroutine blocks, your program blocks.
The argument between whether to choose threads or coroutines is an argument whether it is better to have blocking or race conditions.
So, having traversed "why I'm wrong", let's consider if there are better models.
Arguably the best language in the class of highly concurrent ones is Erlang. Erlang for years dominated the space of stable concurrency.
And how it works? Message passing. You can still deadlock Erlang program - it's just much harder.
Another good example I like is Grand Central Dispatch and it's OS version libdispatch.
The whole point there is to have blocks or operations that, and this is guaranteed by the programmer, mutate the state of memory in an atomic way.
Both of those examples are really about having a queue of changes and applying them. And you could, quite sensibly, argue that a mutex is nothing more internally than an ordered queue, so why bother.
Well, my problem is really with the API language gives to the programmer. The point I'm trying to make is that concerning the programmer with how his work is executed is wrong.
If we can avoid locks inside Python interpreter, and we can, by having a different syntax to describe what needs to happen, I'd consider it a much better solution.
It's much easier to say: "drive me to the Station" than to drive there.
That is why Redis talks honestly about what the potential problems and strengths are. Those aren't problems in the right use case, but they are things you need to know to help figure out if you have the right use case.
Moving on, cooperative multitasking is a form of cooperative concurrency. It has all of the same strengths and weaknesses as cooperative concurrency in any other setting. Hence a single badly coded network call could freeze up Mac OS 9, or Win 3.1. That said if every application is coded correctly, then you can do a lot of things concurrently. I remember using email, Usenet and a browser at the same time on Mac OS 9, and it worked fine..most of the time.
Once the system gets complex enough, the fact that any mistake can lock everything becomes unacceptable. This is why operating systems do not generally use cooperative multitasking. An individual application may reasonably choose to go either way. But as they get more complex, there is pressure to go preemptive.
Moving on, you only have half the story for Erlang. Erlang uses message passing AND immutable memory. The fact that you can't modify memory in place is a huge limitation on the programmer. However it also eliminates large classes of race conditions.
If Python made all data immutable, it could also get rid of most of the uses of the GIL.
For a less extreme approach, look at Go. In Go data is mutable. But by convention, you pass objects around using channels, and only one goroutine owns an object at a time. Since only the owner accesses it, that eliminates most race conditions. However if you violate the convention, a Go program dump can dump core very quickly!
One exception is Redis/Node.js - I don't like them because they indeed work in certain cases, but that set is much less than what authors of Redis/Node.js advertise.
I very fondly, for example, remember antirez being adamant that Redis is awesome as cache, yet it took me 10minutes to prove that single Redis performs worse than single Memcached. And no, I don't consider answer: run more Redis instances valid. You design software to have features that solve problems, not replace old with new.
And yes, Erlang uses message passing and has locks and immutable memory.
I prefer the Go approach of mutating through channels/queues. Both Go and GCD in C/ObjC are good examples of this approach.
Know what each tool is good for, and don't believe unwarranted hype.
And yes, coroutines have their place. Python without generators would be a much weaker language.
In regard to Python - generators make sense, coroutines, much less. That's my opinion, but over the span of 10 years most programmers I met professionally either never used coroutines outside of pet projects or use them sparingly.
I'd actually like to see a big project with coroutines implemented using the async/await.
Take working in Tornado. I've been writing in Tornado, and all I've needed to do is decorate functions with @gen.coroutine then use generator syntax. All the detailed plumbing to implement the event loop is done for me, and I never interact with it directly. Nor would there be any value in my doing so. It needs to be implemented once, and implemented well. Once that is done there is little to no value in messing with it. You'd just be creating the opportunity for disaster with little to no corresponding value.
I actually experienced this exact disaster. In this project, Tornado has to interact with Redis. So I used tornado-redis which tried to be implemented directly. Unfortunately it was not well implemented, and one coroutine could easily get a message meant for another. Untangling the mess promised to be a lot of work.
However it was literally the work of minutes to switch my class to the synchronous Redis library, write a ThreadPoolExecutor wrapper for my class, and have Tornado interact with that. It would be slightly better if the native version had worked, but this got 95% of the benefit for under 1% of the work.
You could describe this as "using coroutines sparingly". I'd describe it as using them sensibly. Coroutines are a useful bit of plumbing to enable a programming style, but their flexibility is also a burden. You should use them to build a sensible abstraction, then use that.
For another example of this, look at Scheme. Scheme internally implements continuations, and function calls are just a special case of that. Continuations are, of course, a building block for coroutines, generators, and many other programming constructs.
However a sensible Scheme programmer just implements functions. They may be using a project that somewhere does coroutines, continuations, and all sorts of other fun stuff. But the day to day code you write doesn't do that. There is no need to add the mental overhead from using the construct all of the time "just because it is there".
First of all, when I'm wrote this article by coroutines I meant specifically Python async/await pattern which I don't find easy to use or read.
And I agree on Tornado. Over the years I've seen many APIs that abstract away stuff like implementation details of cooperation yet are in fact coroutines - e.g. goroutines in Go, tasklets in Stackless. Even Erlangs processes are, on the VM level, cooperatively scheduled green threads.
For me there are 3 problems when you design concurrency APIs. One is performance and language internals. Second is API that you expose to the end programmer. Third problem is what you can achieve as a programmer using those API based on how integrated concurrency is into the language.
All the examples of higher level platforms I know are good for highly concurrent apps: Erlang, Go, Node.js, GCD - they were designed and throughly integrated into the language.
In that context threading in Python really feels bolted on.
And for something different: seeing how some people reacted I'm diving deep to make 2-3 articles of tour-de-concurrency. What do you think?
Cooperative multitasking is easier in regard to design of OS or language that implements it - but honestly, almost all production code today is glue logic and almost every app is big and messy - e.g. XCode takes ~150MB just to start on my Mac
That said, there are many possible trade-offs between preemption and cooperation. It is old, but http://www.bitmover.com/llnl/smp.pdf made a good case for why SMP was the wrong way to scale to large numbers of CPUs. NUMA is a much better architecture. SGI demonstrated this in the 1990s with 1000 CPU machines. Unfortunately Linux took the SMP path like Solaris before it, and our performance is suffering as a result. :-(
An interesting historical note. Larry McVoy's thinking about how to handle contention lead him to thinking about a very, very slowed down version of it - distributed source control. This lead to a "side project" named BitKeeper. Move a few years later and that was a real product used for the Linux kernel. After some drama, he withdrew permission for Linux to continue using it for free and in response Linus wrote git. (Linus and Larry were always on good terms. The drama was between Larry and Jeremy Allison, of SAMBA fame.)
Of course people do use coroutines - I simple don't think they're a good idea. I think they're a bad idea. They're neither readable nor introduce better performance.
And yes, I really don't think people think they need concurrency - they just want this_A and this_B to happen ASAP and with no blocking.
On your explanation of coroutines and preemptive - I agree, though I consider both equally bad.
The point I was making that Celery runs lockless - and that'd be an interesting idea to back port to Python.
And I know queues are old idea. What I like about them is the ability to get rid of "locks everywhere". An example of what I consider good concurrency model is Grand Central Dispatch with atomic ops running via queues.