I was inconvenienced by multiprocessing just today: We were using it to spin up a bunch of workers, and have spent some time debugging an issue with SQLAlchemy, which requires special management when forking: "it’s usually required that a separate Engine be used for each child process. This is because the Engine maintains a reference to a connection pool that ultimately references DBAPI connections - these tend to not be portable across process boundaries." [1]

So you have to be careful when dealing with forking and databases, which has lead us to subclass multiprocessing.Process specifically for our (common) case of wanting to continue to use the session object in the child process without having to think about recycling the Engine object. It was that code we have been trying to debug today, because in some specific cases we still run into issues (yes, we read the docs). Also, when people (usually new engineers) don't know about this they usually blindly use the multiprocessing module directly (and who can blame them) and end up spend some time debugging intermittent connectivity issues until someone says "oh, you should use the xyz module, it handles the DB stuff for you".

[1] http://docs.sqlalchemy.org/en/rel_0_9/core/connections.html#...

You probably know this already, but if you're using (or are amenable to using) uwsgi for managing your workers, there's an option[1] that can control whether your application code is loaded before or after fork() is called.

I've run into the exact issue you've described; the --lazy-apps solution may be slightly more resource intensive (since you maintain N copies of the application in memory), but ends up being much nicer to reason about.

1. http://uwsgi-docs.readthedocs.org/en/latest/articles/TheArtO...

>it’s usually required that a separate Engine be used for each child process.

Yeah, provided you do this there should be no headache.

>So you have to be careful when dealing with forking and databases, which has lead us to subclass multiprocessing.Process specifically for our (common) case of wanting to continue to use the session object in the child process without having to think about recycling the Engine object.

What use case led to this being a non-negotiable requirement?

It's not non-negotiable, it's for safety and convenience and DRY: We want our engineers to be able to use the database in their parallel processing jobs without having to understand the internals of SQLAlchemy, Engines, and forking.

multiprocessing can be fickle. In ipython, ctrl-c during a computation using multiprocessing results in general unhappiness, and will probably be followed by a 'killall python'

I've had this problem too, but after doing the (quite considerable) leg-work to fix it involving messing around with process groups and the ridiculous number of different types of signal, I'm more inclined to blame the horrible 30 year old UNIX APIs than python.

I've had this signal handling problem in single process/single threaded code too, and in other languages.

True, it would be nice if python layered on a nicer set of APIs to deal with the nastiness.

>I'm more inclined to blame the horrible 30 year old UNIX APIs than python.

Datapoint: I encountered the same thing on Windows, having to use the Powershell equivalent of `killall python` to deal with it.

No, this is pretty easy in c. i think the python docs even apologise for it somewhere.

Oh, yeah - and this one! Much more so than the inconvenience I posted!

I gave-up on multiprocessing a long time ago. I was happily writing code that used it on linux and then tried running it on FreeBSD. Turned-out that it worked basically because of nuances of glibc but was totally not fork safe. The most simple things broke. I don't know if they have ever been addressed.

Other major inconveniences I've run into:

Collecting metrics require using an external collector process (statsd) rather than just an in-proc lib (e.g., codahale metrics on the JVM).

Issues generating random numbers - I ran 8M simulations on 8 cores, but because the random seed was forked, I actually only ran 1M simulations 8 times. (Easy enough to fix, but annoying.)

Deduplication is really handy in reducing memory usage - why would you want the same (immutable) object duplicated N times?

And of course various issues with resources - you need to be careful only to connect to the DB/etc after forking otherwise stuff gets weird. And of course, this means that instead of sharing a single connection pool, you've now got one DB connection per proc, which may or may not be a big waste of resources.

> Issues generating random numbers - I ran 8M simulations on 8 cores, but because the random seed was forked, I actually only ran 1M simulations 8 times. (Easy enough to fix, but annoying.)

IIRC, even if you do your easy fix of giving the 8 threads different seeds, your results are not trustworthy since the parallell random number streams are likely to exhibit significant correlations. There's quite a bit of algorithm research behind quality PRNGs.

I have. I've personally worked on a project that had to be significantly refactored because the original author didn't understand the implications of GIL.