News for hackers. Stuff that matters.
Re:The multiprocessing module in Python by cossatot on Friday February 03, 2017 @04:51PM (#13563652)
Some good alternatives, for when you just want quick and easy parallelization:

Joblib[0]: 'Embarrassingly parallel for loops'. Basically just write a generator and get multicore processing on it. Pretty straightforward.

Multiprocess[1]: This is a fork of the multiprocessing module. The biggest benefit to me (the one time I used it) is that it's easier to create shared data structures for multiprocessing (which I couldn't do with multiprocessing.Pool). Last week I had to do a really long graph calculation that only needed to return a result for a very, very small fraction of the arguments. Joblib stored all of the null results as 'None', which tore through my RAM after billions of relatively fast calculations before crashing. But using a shared dictionary and multiprocess.Pool and imap_unordered, I was able to use mutlicore processing, adding items to the dict only if the right conditions were met, and discarding the 'None's. RAM use was very minimal.

[0]: https://pythonhosted.org/joblib/index.html [1]: https://pypi.python.org/pypi/multiprocess

Re:The multiprocessing module in Python by d33 on Friday February 03, 2017 @05:21PM (#13563857)
Also, consider using Celery [0] and distribute the computation across different hosts as well.

[0] http://www.celeryproject.org/