News for hackers. Stuff that matters.
Re:The multiprocessing module in Python by rdtsc on Friday February 03, 2017 @02:45PM (#13562629)
There is a nice hack for multiprocessing module to use for IO-bound code without having to fork processes:

   from multiprocessing.dummy import Pool as ThreadPool
   pool = ThreadPool(16)
   res = pool.map(one_arg_fun, [arg1, arg2, ...])
A 3 line IO parallelism speedup trick. Used it fetch stuff from multiple servers recently.
Re:The multiprocessing module in Python by gshulegaard on Friday February 03, 2017 @03:25PM (#13562988)
I would recommend you use coroutines rather than multiprocessing.dummy for balancing blocking operations (like IO). I believe multiprocessing.dummy is just a wrapper for threading which has context switching overhead [1].

Python3 has asyncio as part of the stdlib but for Python2 there are a variety of coroutine libraries out there (I think the most widely recognized being gevent).

[1] https://www.youtube.com/watch?v=Obt-vMVdM8s

Re:The multiprocessing module in Python by rdtsc on Friday February 03, 2017 @04:06PM (#13563304)
Switching code-base to Python 3 or installing gevent/eventlet is certainly more than a 3 line hack.

> I believe multiprocessing.dummy is just a wrapper for threading which has context switching overhead [1].

Yap. But I am fetching and waiting for data to come from servers half-way across the country. Not worried about too much thread switching overhead. The point it was still faster than a for loop with a fetch. It was replaced by 3 lines which was 10x faster.

Re:The multiprocessing module in Python by bogomipz on Friday February 03, 2017 @03:42PM (#13563128)
I'm not following. Can you explain the hack? Python releases the GIL during I/O right, isn't that desirable?
Re:The multiprocessing module in Python by rdtsc on Friday February 03, 2017 @04:02PM (#13563276)
> Python releases the GIL during I/O right, isn't that desirable?

Python releases GIL during IO so that's why thread-based pool works for IO parallelism and there is no need to fork. Was that what you mentioned? Sorry, I don't think I completely followed.

Re:The multiprocessing module in Python by bogomipz on Friday February 03, 2017 @07:44PM (#13564781)
I just re-read your comment again and the one above it. I was confusing MT and MP. Thanks.
Re:The multiprocessing module in Python by ipunchghosts on Friday February 03, 2017 @03:07PM (#13562831)
can you elaborate?
Re:The multiprocessing module in Python by jessedhillon on Friday February 03, 2017 @03:33PM (#13563050)
It's the equivalent of: (edit: actually an improvement over)

    from threading import Thread


    def blocking_function(arg):
        ....


    pool = []
    for arg in args[:16]:
        f = lambda: blocking_function(arg)
        thread = Thread(target=f)
        thread.start()
        pool.append(thread)

Edit: see below, this doesn't include job queueing but rather, hard limits your input to maximum 16 args. The point is, the code is a nice, easy way to start a number of threads working on a list of arguments.
Re:The multiprocessing module in Python by quinnftw on Friday February 03, 2017 @04:00PM (#13563267)
Not quite. You can submit as many jobs as you'd like to the pool and it will delegate them to 16 threads. The pool maintains a queue of work to be done and the fixed number of threads consume from the queue.
Re:The multiprocessing module in Python by rdtsc on Friday February 03, 2017 @04:08PM (#13563320)
Almost but as quinnftw mentioned it does a bit more, specifically it limits the number of threads it runs to 16. I can pass 1000 url or arguments into it and it won't spawn 1000 threads but keep it at 16 max. Also I think it does a bit a better job with exception propagation.
Re:The multiprocessing module in Python by paulddraper on Friday February 03, 2017 @04:08PM (#13563324)
Multiprocessing offers a ThreadPool for multi-threading..

It doesn't have true CPU parallelism like multiple processes would have, but it works for IO code.

It's a little gold nuggest in the multiprocessing lib.