I really hate this argument for its disingenuity. Some things (including message passing) are naturally fastest with shared memory. Multiple processes are not an equivalent substitute.
Clojure, for instance, leverages shared memory to implement its excellent, lightweight STM and related high-level threading constructs, and shared memory is what allows Clojure to easily implement cheap, MVCC functional data structures.
Multiple processes aren't an answer, here. To implement the same high-level constructs efficiently requires re-introducing shared memory through more complex, less efficient mechanisms, and often leaves the problem of sharing access to higher-level constructs (objects, instead of data) unsolved.
Multiple processes are a poor work-around for a lack of support for concurrency, not a solution. They make sense if you're sandboxing, but make little sense for implementing a high-performance concurrent server.
To pre-empt the erlang discussion -- erlang's message passing model does scale, but Erlang's runtime does require support for real shared memory concurrency in order to maximize single-machine performance.
This isn't a disingenuous answer at all, since it goes on to point out that A Web app is one giant workload where this model makes sense: multiprocessing is the approach you should be taking with a Web application.
As someone with a strong Java background, I have to continually force myself to be aware of my bias against concurrency-via-multiple-processes.
If you have a stateless application (and if at all possible, statelessness is a good thing [1]), then multiple processes scale vertically reasonably well (on the same machine), but having a multiple-process architecture makes scaling horizontally (across machines) natural and easy.
It's true that shared access to higher-level constructs in a multiple process environment isn't solved, but (a) shared access to data is an anti-pattern [2], and (b) Memcache plus some kind of serialization works pretty well when you do need shared data.
[1] Yes, applications exist where statelessness doesn't make sense. In this case, though the author was originally developing on AppEngine, so I doubt there is much state in the app.
[2] I'm not saying you should never have shared data. I am saying you should avoid it as much as possible.
Everything from database connections to user data can be cached locally, shared across connections, and does not require multiple processes each with their own large heap.
I'd like to elaborate on more examples (such as comet and efficient handling of a large number of blocked connections while allowing unblocked connections to proceed concurrently while maintaining cached local state) ... But I'm on an iPad and this keyboard is driving me crazy.
But I think caching is something best done in system/platform code, not in your application.
Like you say, there are plenty of examples where sharing state makes sense. But I still believe that state in application code is something that be avoided, and that most of the time it's best to rely on platforms that do state management for you.
For example, session support in web platforms is a great example of something that supplies (simulated) state, and is supplied by the platform.
I have a hard time with the notion that web apps should be "stateless." It seems to be an argument borne out of limitations of the frameworks/platforms being used, and repudiated by the fact that such platforms do share state, but are forced to use less efficient external mechanisms (such as network requests to memcached, the local database, etc), rather than leveraging the advantages of data locality as available in a non-multiprocess system.
We've often taken advantage of sticky sessions to allow individual servers to maintain and share state across requests while ensuring that the state could be reconstructed by another server should it become necessary. It is simply more efficient to do so, and efficiency in implementation directly translates to dollars spent on operational costs, as well as effects on human observable response times.
But I think it's an illuminating discussion anyway.
I totally agree with you on the point regarding worrying about delivering a product instead of rewiting the work. That's why the services already written will remain in place and I'll there fore be running Python alongside the Clojure/JVM stack.
Yikes, really? That sounds even worse than a clean break. You don't shed your perceived problems with Python and instead get to manage two codebases and stacks. Your operations team will absolutely love you in the future.
Based upon a casual read of what your service does, I'm really stressing to think of a CPU-bound situation that can't be pooled into a multiprocessing pool. All of the heavy work I can think of your service doing -- particularly crawling -- is going to end up network or I/O-bound, isn't it? I'm coming up empty and working with a theory. Perhaps you can share what your CPU-bound process is?
Of course most things can bee pooled into a multiprocessing pool. It's just a matter of the pain you have to endure while doing so. For some tasks it's a straight forward process while for other tasks it can be pretty painful. The crawler is IO-bound but most of the collaborative filtering algorithms are CPU-bound and having nice concurrency constructions in the language is a bonus.
And if that's too heavy, there's always stuff like GEvent. Asynchronous I/O and multiprocessing solves 80% of all problems you may have.
I do wish that Python / Ruby would be GIL-free, but for my startup I choose these platforms because they've got no parallel when it comes to rapid-development of web applications, because no matter how cool this and that language is, nothing beats the productivity gained by robust, battle-tested web frameworks (not to mention other stuff, like NLTK or SciPy).
As an early-adopter myself I might try out a couple of projects in Clojure, but I wouldn't bet my business on it unless I saw a clear need that Clojure satisfies and that dwarfs all other disadvantages, like platform immaturity.
Saying that Asynchronous I/O and multiprocessing solves 80% of the problems is hand-waving away the complexity such approaches might add to a project. Clojure simply gives you more and better tools then Ruby and Python currently do for dealing with concurrency.
You might not bet your business on Clojure, but certainly other people have for a couple years now and they don't seem to take issue with it.
EDIT: removed the defensive bit. My experience, as this thread shows, is that people get touchy about the limitations of the GIL.
EDIT: RE: Don't take me wrong, I sometimes hate the limitations that the GIL brings.
I'm continually looking at alternative implementations, like Pypy or Rubinius or JRuby, or other languages, like Haskell or Clojure, but then I end up doing a lot of yak shaving and nothing gets done.
You say that like it's a bad thing. ;)
As for the GIL, the author didn't even consider multiple processes -- he explained them away as a solution for some workloads. A Web app is one giant workload where this model makes sense: multiprocessing is the approach you should be taking with a Web application. Tie a request to one core, and spawn enough WSGI applications for the number of cores you have (and then some). That's an elegant solution to this problem, since a request doesn't need to fart around with other requests in most cases. If your app can't handle multiple copies of itself running, how do you expect it to scale?
This reads a little bit like a waffling, like a person who considers a completely different environment and rewrite of the entire stack a solution to some thorns in Python. I'm all for picking an alternative Web stack, but:
> I had written a database loader to import Apple’s Enterprise Partner Feed (EPF) and a web crawler in Python and next up was the web interface.
> It all seemed like a smooth sailing but in the back of my head I was beginning to have doubts about my decisions.
> Why we are choosing Clojure as our main programming language
This being the third blog post for an as-yet-unreleased product, I'd say worry about delivering a product instead of justifying a rewrite of your work thus far on your blog. If I were considering funding your startup, this blog post would be a fairly bad sign to me.
At any rate, a computer language is just a tool to implement an idea, and focusing strongly on the language of choice is busywork itself.