non-shared shared
compute-bound multiprocessing ???
I/O-bound either coroutines
I put ??? in the upper right because as far as I know Python doesn't have any general solutions here.Also, if "most tasks you've seen" are compute-bound at some points, then of course it makes sense to use Celery! But you should bear in mind that other people may have seen other kinds of task.
And it's not that I think Celery is the best. It has huge drawbacks.
I merely suggesting that the model Celery and GCD enforce is much better for writing CPU, I/O and CPU+I/O bound code.
Well, I need a system that is responsible for maintaining the shared data, and I have to program my tasks to send it queries and updates, and receive the results, and so probably I'm going to need serializers and deserializers for all my data structures, and if the tasks have transactions (which they almost certainly do) then I'm going to have to implement locks to maintain the data integrity. Whatever it is, this system looks a lot like a database.
Whereas with coroutines each task just updates ordinary data structures in memory. No need for communication, queries, serialization, or locks.
Queue and Thread is available in python and work well up to a point. Twisted (and sugar @inlineCallbacks) or similar makes for much more concise code.
Reading through the presentations, it appears having proper OS support is also necessary, so it's not just a Python GIL problem.
What I'm trying to advocate is that the distinction of CPU bound and I/O bound makes sense in CS classroom, but most of the tasks in production code are a mix of both.
Therefore, the distinction is really between blocking and non-blocking, and Celery enables "almost readable" non-blocking coding