Re:The multiprocessing module in Python
by jessedhillon on Friday February 03, 2017 @03:33PM
(#13563050)
It's the equivalent of: (edit: actually an improvement over)
Edit: see below, this doesn't include job queueing but rather, hard limits your input to maximum 16 args. The point is, the code is a nice, easy way to start a number of threads working on a list of arguments.
[ Parent
| on HN ]
Re:The multiprocessing module in Python
by quinnftw on Friday February 03, 2017 @04:00PM
(#13563267)
Not quite. You can submit as many jobs as you'd like to the pool and it will delegate them to 16 threads. The pool maintains a queue of work to be done and the fixed number of threads consume from the queue.
Almost but as quinnftw mentioned it does a bit more, specifically it limits the number of threads it runs to 16. I can pass 1000 url or arguments into it and it won't spawn 1000 threads but keep it at 16 max. Also I think it does a bit a better job with exception propagation.