News for hackers. Stuff that matters.
Re:Meta pledges Three-Year sponsorship for Python if GIL rem... by tga_d on Saturday July 08, 2023 @12:30PM (#36645964)
I have no experience with Windows, but at least on Linux, if the forking is handled intelligently (namely, late enough that identical assets don't need to be re-written), duplicated memory should be handled by Copy on Write.
Re:Meta pledges Three-Year sponsorship for Python if GIL rem... by jmuhlich on Saturday July 08, 2023 @02:43PM (#36647437)
CPython still needs to update reference counts after the fork even if your python code itself doesn't modify any data, making Copy on Write less helpful than it would first appear. Monolithic memory buffers like numpy arrays can be shared efficiently via CoW, but rich data structures with many PyObjects suffer from the refcounting issue.
Re:Meta pledges Three-Year sponsorship for Python if GIL rem... by sanxiyn on Saturday July 08, 2023 @02:50PM (#36647507)
Have you tried it? You are right in general, but even if you don't write anything in Python, CPython runtime constantly writes to heap, so everything is copied. You need specific support from CPython runtime to exploit copy-on-write.

This is documented: https://docs.python.org/3/library/gc.html#gc.freeze

For details, see "Copy-on-write friendly Python garbage collection" by Instagram Engineering: https://instagram-engineering.com/copy-on-write-friendly-pyt...

Re:Meta pledges Three-Year sponsorship for Python if GIL rem... by tga_d on Saturday July 08, 2023 @07:44PM (#36649883)
I guess when GP said "heavy ... C extensions" I was thinking code pages from libraries and structures managed by C, yes python objects will require additional consideration.
Re:Meta pledges Three-Year sponsorship for Python if GIL rem... by gjulianm on Saturday July 08, 2023 @05:34PM (#36649000)
Multiprocessing in Python isn't exactly like forking on other languages. There's no shared Python memory, it's just a new interpreter instance, and then you have to use sharing mechanisms (shared arrays/values, pipes, queues) to transfer data explicitly.
Re:Meta pledges Three-Year sponsorship for Python if GIL rem... by tga_d on Saturday July 08, 2023 @07:45PM (#36649893)
I know, but the entire point of CoW is you don't need shared memory, as long as the fork happens after the memory was written it'll just act like an entirely independent copy transparently to the application, with the OS duplicating the backing page on the first write.

Edit: Ah I see, you're saying the library itself discards anything created from python. That sounds like an implementation problem then, I'd expect it to at least keep imports so any static code pages don't need to be recreated (the interpreter itself should be fine at least, unless it's doing something really dumb).