Re:In C, how do you know if the dynamic allocation succeeded?
by willbudd on Friday November 05, 2021 @09:30PM
(#29126673)
Sounds to me like you (or Python devs?) should be looking at clone(), which is basically a lower level version of fork() that provides precise control over every detail with which the new process is created, including what (if any) parent memory you want to share.
The manual page is quite informative: man clone
Re:In C, how do you know if the dynamic allocation succeeded?
by 01100011 on Saturday November 06, 2021 @08:37AM
(#29129702)
This. When my team hit the issue we just made a subprocess replacement which relied on clone instead. Before that, we would trigger OOM when trying to run external commands while still having 30% free memory.
Re:In C, how do you know if the dynamic allocation succeeded?
by shoo on Friday November 05, 2021 @09:38PM
(#29126727)
it looks like cpython is improving and cpython 3.10 now uses vfork when launching subprocesses in linux
years ago when i first hit this in production, we ended up working around it by rewriting our application code to use a pure-python library that did the equivalent thing as the separate command line tool we were trying to launch.
We didn't actually want to fork anything and share gigabytes of virtual memory with the child process, we wanted to spawn an almost entirely independent process to do something and report results, but that got implemented under the hood by fork.
Spawning processes is one area where Windows is more elegant than linux: windows offers spawn. Apparently macos and solaris implement a posix_spawn that avoid the complications of fork/exec.
linux offers posix_spawn, apparently which may may or may not call fork under the hood depending on which libc you're using. If libc implements posix_spawn by calling fork then you're back in the same mess with linux heuristic memory accounting and overcommit. E.g. old versions of glibc will fork when you posix_spawn, newer versions of glibc may vfork . musl apparently will always vfork.
It looks like cpython's subprocess.Popen was patched in python 3.8 to detect some cases where posix_spawn can be used -- it reads as if it will only kick in on linux if it detects a sufficiently new version of glibc: https://github.com/python/cpython/blob/main/Lib/subprocess.p...
edit: Python 3.10 now supports using vfork for linux inside subprocess: https://bugs.python.org/issue35823
edit 2: here's a similar tale from go, replacing use of fork in fork/exec:https://github.com/golang/go/issues/5838
https://go-review.googlesource.com/c/go/+/37439/
https://about.gitlab.com/blog/2018/01/23/how-a-fix-in-go-19-...