News for hackers. Stuff that matters.

A comment on What's up, Python? The GIL removed, a new compiler, optparse deprecated. Parent comment

Re:What's up, Python? The GIL removed, a new compiler, optpa... by oivey on Sunday July 30, 2023 @09:53PM (#36938004)
If subprocesses die (segfault maybe) it isn't uncommon for them to not be cleaned up and/or cause the parent process to hang while it waits for the zombie to respond. That's one I experienced last week on Python 3.9. A thread that experienced that would likely kill the parent process or maybe even exit with a stacktrace. Way easier to debug, and doesn't require me to search through running tasks and manually kill them after each debug cycle.

My impression is that the multiprocessing module is a heroic effort, but unfortunately making the whole system work transparently across multiple OSs and architectures is a nearly insurmountable problem.

[ Parent | on HN ]
Re:What's up, Python? The GIL removed, a new compiler, optpa... by empthought on Sunday July 30, 2023 @10:17PM (#36938125)
You may be interested in the concurrent.futures library, available for over a decade now. It keeps you from shooting yourself in the foot like that.

https://docs.python.org/3/library/concurrent.futures.html

[ Parent | on HN ]
Re:What's up, Python? The GIL removed, a new compiler, optpa... by KolenCh on Sunday July 30, 2023 @11:38PM (#36938571)
Why do you think it would help?

It provides a nice interface but is using multiprocessing or multi threading under the hood depending on which executioner you use:

> The ProcessPoolExecutor class is an Executor subclass that uses a pool of processes to execute calls asynchronously. ProcessPoolExecutor uses the multiprocessing module, which allows it to side-step the Global Interpreter Lock but also means that only picklable objects can be executed and returned.

[ Parent | on HN ]
Re:What's up, Python? The GIL removed, a new compiler, optpa... by empthought on Monday July 31, 2023 @12:30AM (#36938830)
Your trouble seems to involve not understanding how to set up signal handlers, which ProcessPoolExecutor handles for you and exposes via a BrokenProcessPool exception.
[ Parent | on HN ]
Re:What's up, Python? The GIL removed, a new compiler, optpa... by wiseowise on Monday July 31, 2023 @03:01AM (#36939571)
> Derived from BrokenExecutor (formerly RuntimeError), this exception class is raised when one of the workers of a ProcessPoolExecutor has terminated in a non-clean fashion (for example, if it was killed from the outside).

What if it hangs?

[ Parent | on HN ]
Re:What's up, Python? The GIL removed, a new compiler, optpa... by empthought on Monday July 31, 2023 @06:08AM (#36940577)
That isn’t the scenario originally described, but there is a timeout parameter in future.result().
[ Parent | on HN ]
Re:What's up, Python? The GIL removed, a new compiler, optpa... by nine_k on Monday July 31, 2023 @12:19AM (#36938766)
Always setting a timeout on every IPC or network operation helps immensely. IIRC multiprocessing module allows that everywhere, but defaults to waiting forever in a couple of places.
[ Parent | on HN ]
Re:What's up, Python? The GIL removed, a new compiler, optpa... by emmelaich on Monday July 31, 2023 @10:57PM (#36951790)
Zombies don't respond, they merely have to be wait()'d for. Which should take microseconds at most.

I've seen orphaned processes sometimes idle, sometimes busy doing god knows what. But Zombies OTOH are rarely a problem, and should be able to be dealt with easily.

Perhaps the desire of Python to be Windows compatible mitigates against some design more suitable for Unix.

[ Parent | on HN ]