News for hackers. Stuff that matters.
Re:No-GIL mode coming for Python by bjourne on Friday July 28, 2023 @09:21PM (#36915747)
This can (and I think will) cause issues for C extensions because many are written without multi-threading in mind. Here is a small example which is unsafe if lst can be accessed from another thread: https://news.ycombinator.com/item?id=36649769 Note that the code may cause a context switch even today if the C code callbacks into Python bytecode (via a __del__ method) and the bytecode is long enough (100 instructions I think). However, that is extremely unlikely and much C extension code is not written with such situations in mind.

People using C extensions may also rely on them executing atomically. For example, you could have a thread pool that posts and receives from a numpy array. Would work fine today but break without the GIL.

Re:No-GIL mode coming for Python by qbasic_forever on Friday July 28, 2023 @09:26PM (#36915786)
Yep there are a ton of issues like that to be found, and unfortunately they will manifest as difficult to find and debug race conditions. This is why the proposal and work is to make non-GIL mode entirely optional and not the default.

It just means for the brave few that flip it on and use it, be prepared to spend a huge amount of time finding and fixing subtle race conditions in decades of old python library code. The early adopters are going to be in for a lot of pain, or more likely they'll restrict their use of non-GIL processes to very specialized and dedicated processes that have as few dependencies as possible.

Re:No-GIL mode coming for Python by miraculixx on Friday July 28, 2023 @09:47PM (#36915934)
The intent is to make no GIL the default eventually.
Re:No-GIL mode coming for Python by n2d4 on Friday July 28, 2023 @10:08PM (#36916089)
I don't think this is true. There are fairly strong voices on both sides inside the community, at this time it's pretty uncertain.

To quote Guido:

>Let’s not blow it this time. If we’re going forward with nogil (and I’m not saying we are, but I can’t exclude it), let’s make sure there is a way to be able to import extensions requiring the GIL in a nogil interpreter without any additional shenanigans

https://discuss.python.org/t/pep-703-making-the-global-inter...

Re:No-GIL mode coming for Python by Alphaeus on Saturday July 29, 2023 @12:41AM (#36917010)
The Steering Council said their intention is to remove the GIL-build in future:

> Long-term, we want no-GIL to be the default, and to remove any vestiges of the GIL (without unnecessarily breaking backward compatibility). We don’t want to wait too long with this, because having two common build modes may be a heavy burden on the community (as, for example, it can double test resources and debugging scenarios), but we can’t rush it either. We think it may take as much as five years to get to this stage.

https://discuss.python.org/t/a-steering-council-notice-about...

Re:No-GIL mode coming for Python by toyg on Saturday July 29, 2023 @04:00AM (#36917967)
By the usual laws of software estimates, if they think it will take 5 years, it's going to be more like 10-15.
Re:No-GIL mode coming for Python by hinkley on Saturday July 29, 2023 @12:29AM (#36916931)
Does Python have a lot of secondary dependencies? I could see someone pulling in two dependencies, not realize they both use the same unsafe library, and end up having them step all over each other.
Re:No-GIL mode coming for Python by qbasic_forever on Saturday July 29, 2023 @12:35AM (#36916967)
It does, so much so things like virtualenv were introduced so every program can have its own set of dependencies such that they won't clash with other libs on your system. Something like flask or fastapi pull in a lot of secondary dependencies alone.
Re:No-GIL mode coming for Python by hinkley on Saturday July 29, 2023 @04:51PM (#36924424)
Probably doesn't work with FFI though? I'm trying to get a sense of how hypothetical versus very real this problem is.
Re:No-GIL mode coming for Python by smcl on Saturday July 29, 2023 @04:51AM (#36918270)
I think the CPython core devs are very keenly aware of these issues. Otherwise they'd have announced a plan to suddenly rip out the GIL altogether, rather than a phased approach that allows people to opt-in to the no-GIL mode.
Re:No-GIL mode coming for Python by jillesvangurp on Saturday July 29, 2023 @05:41AM (#36918505)
That's fine. We go from suspecting there are issues to knowing exactly where the issues are. The rest is just chipping away at that list and making the issues go away. Either by adding some kind of mutex around the code or by replacing the native code with something less likely to have issues.

The argument against this seems mainly that it's a lot of work; not that it's impossible work. It probably is a lot of work but if there are enough people doing the work, we should get some results.

Re:No-GIL mode coming for Python by mkoubaa on Saturday July 29, 2023 @07:29AM (#36919164)
Might be a good time to write concurrency first C extension to compete with the legacy ones
Re:No-GIL mode coming for Python by xmcqdpt2 on Saturday July 29, 2023 @07:44AM (#36919257)
> Note that the code may cause a context switch even today if the C code callbacks into Python bytecode (via a __del__ method) and the bytecode is long enough (100 instructions I think). However, that is extremely unlikely and much C extension code is not written with such situations in mind.

As someone who works professionally on a parallel / async runtime that supports thousands of continuously running servers, "extremely unlikely" means that actually it breaks all the time, but it's also impossible to debug.