News for hackers. Stuff that matters.
Re:Meta pledges Three-Year sponsorship for Python if GIL rem... by hgs3 on Saturday July 08, 2023 @11:42AM (#36645443)
PEP 703 mentions the multiprocessing module which I used recently with great success. It is essentially message passing parallelism, i.e. the actor model. I would think even with the GIL removed message passing is still the best approach for a higher-level language, like Python, to handle parallelism as it is conceptually simple and lock free.

The primary disadvantage of the multiprocessing module is it uses separate processes which means my resource heavy Python extension needs to be duplicated multiple times in memory (at least on Windows). I think the move to supporting threads means the module could leverage them instead of separate processes, avoiding the need to duplicate C extensions in memory, and thereby reducing the overall memory pressure on a system.

Re:Meta pledges Three-Year sponsorship for Python if GIL rem... by packetlost on Saturday July 08, 2023 @12:12PM (#36645754)
I agree that message passing can be a really good choice. The problem is the overhead of passing large messages can be really substantial. In general, you'll need to store messages in memory 4x per message (at least temporarily): once on each end for the Python representation and the serialized "on the wire" representation. There's also the issue with `pickle` generally being a horrible serialization format and being the default.

There are ways around this, Python has a rather rudimentary and archaic shared memory subsystem[0] that can make this much more efficient.

On the other hand, the Python multiproccessing sync primitives actually work over a network with almost no code changes, so you can horizontally scale with little effort... but then you're paying the serialization and network overhead, so ymmv.

[0]: https://docs.python.org/3/library/multiprocessing.shared_mem...

Re:Meta pledges Three-Year sponsorship for Python if GIL rem... by tga_d on Saturday July 08, 2023 @12:30PM (#36645964)
I have no experience with Windows, but at least on Linux, if the forking is handled intelligently (namely, late enough that identical assets don't need to be re-written), duplicated memory should be handled by Copy on Write.
Re:Meta pledges Three-Year sponsorship for Python if GIL rem... by jmuhlich on Saturday July 08, 2023 @02:43PM (#36647437)
CPython still needs to update reference counts after the fork even if your python code itself doesn't modify any data, making Copy on Write less helpful than it would first appear. Monolithic memory buffers like numpy arrays can be shared efficiently via CoW, but rich data structures with many PyObjects suffer from the refcounting issue.
Re:Meta pledges Three-Year sponsorship for Python if GIL rem... by sanxiyn on Saturday July 08, 2023 @02:50PM (#36647507)
Have you tried it? You are right in general, but even if you don't write anything in Python, CPython runtime constantly writes to heap, so everything is copied. You need specific support from CPython runtime to exploit copy-on-write.

This is documented: https://docs.python.org/3/library/gc.html#gc.freeze

For details, see "Copy-on-write friendly Python garbage collection" by Instagram Engineering: https://instagram-engineering.com/copy-on-write-friendly-pyt...

Re:Meta pledges Three-Year sponsorship for Python if GIL rem... by tga_d on Saturday July 08, 2023 @07:44PM (#36649883)
I guess when GP said "heavy ... C extensions" I was thinking code pages from libraries and structures managed by C, yes python objects will require additional consideration.
Re:Meta pledges Three-Year sponsorship for Python if GIL rem... by gjulianm on Saturday July 08, 2023 @05:34PM (#36649000)
Multiprocessing in Python isn't exactly like forking on other languages. There's no shared Python memory, it's just a new interpreter instance, and then you have to use sharing mechanisms (shared arrays/values, pipes, queues) to transfer data explicitly.
Re:Meta pledges Three-Year sponsorship for Python if GIL rem... by tga_d on Saturday July 08, 2023 @07:45PM (#36649893)
I know, but the entire point of CoW is you don't need shared memory, as long as the fork happens after the memory was written it'll just act like an entirely independent copy transparently to the application, with the OS duplicating the backing page on the first write.

Edit: Ah I see, you're saying the library itself discards anything created from python. That sounds like an implementation problem then, I'd expect it to at least keep imports so any static code pages don't need to be recreated (the interpreter itself should be fine at least, unless it's doing something really dumb).

Re:Meta pledges Three-Year sponsorship for Python if GIL rem... by shepardrtc on Saturday July 08, 2023 @12:16PM (#36645814)
Multiprocessing is great. Very easy to use and with the Manager(), passing data is easy enough, though very "slow" relatively speaking. It has to serialize the data, send it to the Manager process, then send to all the processes (I believe), and then deserialize it. Perhaps this nogil update can speed that up?
Re:Meta pledges Three-Year sponsorship for Python if GIL rem... by gjulianm on Saturday July 08, 2023 @05:42PM (#36649062)
My experience with the multiprocessing module is exactly opposite. It's been a few years since I last used it seriously (I still use Python a lot) mainly because of how bad it was. The main issue is precisely communication between processes: Python uses the pickle module to serialize/deserialize data, which not only adds quite a bit of performance cost, but it's very hard to debug when for some reason some object is not supported by pickle (and it's a common occurrence).