Re:The multiprocessing module in Python
by ipunchghosts on Friday February 03, 2017 @03:07PM
(#13562831)
can you elaborate?
[ Parent
| on HN ]
Re:The multiprocessing module in Python
by jessedhillon on Friday February 03, 2017 @03:33PM
(#13563050)
It's the equivalent of: (edit: actually an improvement over)
from threading import Thread
def blocking_function(arg):
....
pool = []
for arg in args[:16]:
f = lambda: blocking_function(arg)
thread = Thread(target=f)
thread.start()
pool.append(thread)
Edit: see below, this doesn't include job queueing but rather, hard limits your input to maximum 16 args. The point is, the code is a nice, easy way to start a number of threads working on a list of arguments.
Re:The multiprocessing module in Python
by quinnftw on Friday February 03, 2017 @04:00PM
(#13563267)
Not quite. You can submit as many jobs as you'd like to the pool and it will delegate them to 16 threads. The pool maintains a queue of work to be done and the fixed number of threads consume from the queue.
Almost but as quinnftw mentioned it does a bit more, specifically it limits the number of threads it runs to 16. I can pass 1000 url or arguments into it and it won't spawn 1000 threads but keep it at 16 max. Also I think it does a bit a better job with exception propagation.
Re:The multiprocessing module in Python
by paulddraper on Friday February 03, 2017 @04:08PM
(#13563324)
Multiprocessing offers a ThreadPool for multi-threading..
It doesn't have true CPU parallelism like multiple processes would have, but it works for IO code.
It's a little gold nuggest in the multiprocessing lib.