Re:Let's Remove the Global Interpreter Lock
by cyphar on Monday August 14, 2017 @10:44AM
(#15008905)
You can't just use functions defined in your tool, you need to create a faux-cli interface in order to run each parallel worker. Also, copying large datasets between processes is not efficient. And also, there are cases where the fan-out approach is not the best way of parallelizing a task, and passing information back up to a parent task is more complicated than necessary.
[ Parent
| on HN ]
Re:Let's Remove the Global Interpreter Lock
by dec0dedab0de on Monday August 14, 2017 @11:05AM
(#15009103)
"You can't just use functions defined in your tool, you need to create a faux-cli interface in order to run each parallel worker."
the multiprocessing library allows you to launch multiple processes using your function definitions. It's almost the same as the multithreading library but does not share data.
It seems the real problem, as you pointed out, is the additional memory. I didn't consider situations where each process would need an identical large data set, instead of just a small chunk to work on.
It gets more interesting when you have a large data set that's required for the computation, but as you compute, you may discover partial solutions that can be cached and used by other workers.
So not only a large read-only data set, but also a read-write cache used by all workers. This sort of thing is relatively easy with threads, but basically impossible with multiprocessing.