which python3
which python3nogil
Having interpreters and packages strewn across the machine is a nightmare. The lack of standard tooling has created a lawlessly dangerous wild west. There are no maps, no guardrails, and you have to beware of the hidden snakes. It goes against the zen of python.
As a counter example, Rust packs everything in hermetically from the start. Python4 [1] could use this as inspiration. Cargo is what package and version management should be, and other languages should adopt its lessons.
[1] Let's make a clean break from Python3 even if we don't need a new version right now.
For this you dont neet .envrc and direnv, as this is handled perfectly fine by peen itself: pyenv local <pyenv / virtualenv name>
https://github.com/pyenv/pyenv
I use direnv because I work with many languages and repos and I don't want each language's version manager linked into my shell's profile. As well, direnv lets me control things besides the language version. Finally, direnv means I don't have to explicitly run any commands to set things up. I just cd to a directory.
Plus, you now have to teach and evangelize your method versus the dozens of others out there. It's crazy town.
The negative thoughts and feelings I once had for PHP are now directed mostly at Python. PHP fixed a lot of its problems over the last decade. Python has picked up considerable baggage in that time. It needs to take the time to do the same cleanup and standardization.
I was describing a workflow that works for me to someone who didn't seem to have found an effective Python workflow in hopes that it can work for them too. I work across a variety of languages and none that I've worked with doesn't have some issue that I can't complain about[1]. I personally don't find Python all that painful to work with (and I've been working with it since 1.5.2), but I understand my experience is not universal.
[1] If it's not the language, it's the dependency manager. If it's not the dependency manager, it's the error handing. If it's not the error handling, it's the build process. If it's not the build process, it's the community. If not the community, the tooling. Etc. I have some languages I like more and some less. Mostly it comes down to taste. I'm not here to apologize for or defend Python. I'm only here to describe how I use it effectively, and to correct what I thought were inaccuracies with respect to removing the GIL.
Its insane that nor Node nor Python have a first class version selector.
I don't think it's true. rustup installs new version only when you run `rustup update`. What parent is talking about is pinning a particular rustc version in Cargo.toml, which allows rustup to download that version of rustc to build that particular project/crate.
Consider R, which is filled with the same kind of people. There's one package repository and if your package doesn't build cleanly under the latest version of R, it's removed from the repo.
Don't get me wrong, this has other problems but at least it means that all packages will work with a single language version.
That being said, after two decades of using Python professionally, the only really problems I’ve ever encountered are “package doesn’t support this version for {reasons}” and “ML library is doing something undocumented and/or dumb with a specific Python version.” The former is normally because the package author is no longer their package and the latter is because, again, the ML community is among the absolute worst at creating solid tooling.
However, Python (1991) is only 2 yrs older than R (1993)
That's a vast exaggeration. It is not "really really" hard to spin up a venv and specify your requirements. People just don't do it, and blame the tools for what are bad engineering practices agnostic to any language.
"Really really" not easy would be handling C, C++, etc. dependencies.
I really really wish it was, as then I wouldn't have had to learn Docker.
Maybe more time than getting python deps to work but more deterministic and takes less cleverness.
Similar problem could be with changing behavior for divisions (which actually was more challenging) but similarly you could enable that behavior.
The main problem with migration though was addition of Unicode. You can't just enable it on file by file basis, because once you enable the new behavior in a single file you, will start passing arguments in Unicode to other code in other files and if that code wasn't adapted or will break.
And it was even worse than that because that problem extended to your dependencies as well. Ideally dependencies should be updated first, then your application, but since python 2 was still supported (for a decade after python 3 was released) then there was no motivation to do it.
And if that wasn't enough python 2 already had Unicode support added, but that implementation was incorrect, so even if you imported Unicode_literals from __future__ you potentially broke compatibility with existing python 2 dependencies without guarantee that your code will work on python 3.
IMO that particular change couldn't be done with pragmas, the core issue is that python 3 put a clear separation between text and binary data, but Python 2 mangled them together. That still was true even when you used Unicode in python 2.
The proper way to perform the migration IMO would be to type annotate the code. And then run mypy check in python 3 mode.
But during the time it took to design and deliver Python 3, the language exploded in popularity and reached a much wider audience and 3rd party libraries like numpy became crucial. So when Python 3 was ready it was a completely different ecosystem which was much harder to migrate, but I dont think the core team really relized that before it was too late.
sudo apt install python3ispython
sudo update-alternatives --set python3 /usr/bin/python3-nogil
sudo update-alternatives --set python /usr/bin/python3Python4 needs a hard reset.
venv == virtualenv
virtualenvwrapper is ancient not rly used anymore
pyenv is a third party tool that makes some of this easier, notably around creating more than just a virtual env in that you also choose the Python version.
Python is not hard to deal with in this regard I think people are just uninformed.
The literal only thing you need to understand is “sys.path”. If you inspect this in a shell you will know what you’re up against. Python is all literally just directories. It’s so easy and yet people get so bent out of shape over it.
Create a venv, activate it, and use pip as normal. If you ever run into issues, look at sys.path. That’s it.
And few things have changed since then.
And you can disagree all you want, but it's simply wrong that Python's packaging/venv ecosystem is "just fine".
Is it because people have to use venvs that people complain about it?
I’ll admit being able to install via an OS package manager, vs pip, vs anaconda etc etc can be confusing, but is any of that really Python (the language)’s fault?
pip install
pip install -u
sudo pip install
conda install
sudo conda install
some packages require one, are fine with other with few small warnings, and dont work with third way of installing.
I don’t know the -u flag on pip, never used it can’t find it in the docs.
With a virtual environment sudo is not needed. Assuming you created it, and/or it is owned by you.
Virtual environments are just directories on disk. They are not complex.
I don’t use conda because it’s never felt even remotely necessary to me.
pip and a requirements file is all you need.
how about when you are authoring script under your name, but then want to schedule it for cron to run periodically?
I often find myself working under my user on remote server, but then I want to schedule cron job - and run into all sorts of permissions / bugs and missing packages.
especially when multiple machines need to run this script, and I don't want to involve containers to run 20-lines simple python script.
this is why Golang is so popular - you can just scp a single binary across machines and it will just work.
1. pip install (with/out --user flag )
2. pip3 install (with/out --user flag )
3. sudo pip install
4. sudo pip3 install
5. conda install
6. sudo conda install
/home/you/project/venv/bin/python
As for managing Python library dependencies, I use poetry (https://python-poetry.org), though unfortunately both it and pipenv seem to progressively break functionality over time for some reason.
Because the code that is updated is expected to work with both versions.
> The --disable-gil builds of CPython will still support optionally running with the GIL enabled at runtime (see PYTHONGIL Environment Variable and Py_mod_gil Slot).
Well what py2to3 promised was also not what happened and by far one of worst disasters of a migration in history of open source so you can't blame people for being skeptical
What are you trying to say?
(Not that I disagree with your point, but it’s unclear.)
But I enjoy the irony.
Invoking the concept of “thought terminating cliche”… as a thought terminating cliche.
You literally just replied and we are engaging discussion — empirically the wiki link above has done literally the opposite of what you are saying! At any rate, the intended relevance of the wiki link is the following.
When something bad happens to someone, they tend to overestimate how bad it is relative to something experienced as objectively equally good; it is thus important to be aware of such a bias — in addition to using regular old critical thinking.
In this case: “although this is a ‘migration’, what do we mean when we say that? and is it really the same thing as py2 to py3?”
I would tentatively propose “no” to the latter question; and would additionally propose that calling this a “migration” is not terribly useful as although it’s not incorrect, it’s insufficiently specific and seems to invite a category error.
In addition to the loss aversion bias mentioned above.
Category error: https://en.wikipedia.org/wiki/Category_mistake
"It didn't happen"
Only difference is that they noticed it's a bad idea midway and went back to developing Perl 5
And I still can write "use v5.8" in my Perl script and write script that works in Centos 5 instance from 2007, if I needed to do something on legacy system, and it will work the same on latest Perl.
> because at the time Perl was more popular than Python.
I don't believe that was true at the time as P6 took like decade to decide on semantics and only then the implementations started to happen. More existing code, sure, but much of the web dev moved to PHP/Ruby and many other things to Python.
And unlike Py3 it was initially also much slower than P5, making migration kinda worthless.
I'm not saying for sure this is bad! I do think it is dishonest though about the potential impact. Lots of critical libraries are written in C.
This was the top comment when I added my original comment:
"Fundamentally every single piece of python code ever written will have to stop and now worry about potential race conditions."
https://news.ycombinator.com/item?id=36644114
Most of the other comments here at the time were similarly from people who clearly hadn't read the PEP saying it would break all existing Python code.
So I did my best to represent the backwards compatibility section of the PEP. I told folks to go read it and linked to it. I cited the portion relevant to Python code. There were too many bullet points for the C-API, so I summarized it with the disclaimer "Updating C-API extensions is where the majority of work will be if the PEP is accepted."
I also read the PEP discussion thread where there was disagreement from Python maintainers on how much work would be required of C extension authors, but most of the folks stating it would be a lot of work hadn't seemed to actually have tried to port anything to the new API. Meanwhile Sam had asserted that:
> Most C API extensions don’t require any changes, and for those that do require changes, the changes are small. For example, for “nogil” Python I’m providing binary wheels for ~35 extensions that are slow or difficult to build from source and only about seven projects required code changes (PyTorch, pybind11, Cython, numpy, scikit-learn, viztracer, pyo3). For four of those projects, the code changes have already been contributed and adopted upstream. For comparison, many of those same projects also frequently require changes to support minor releases of CPython.
I don't think I've misrepresented anything.
> No-GIL ain't done until all the C extensions run, especially NumPy.
Sam already got NumPy working without the gil:
* C function called with GIL held
* Extract data needed to do work
* Release GIL
* Do work
* Reacquire GIL
* Modify data, build result
* Return
As an example of the changes, a list could be passed in. I would need some form of locking while processing that list so that mutations while processing won't crash the code.
The GIL does currently result in robust code by default because data can't mutate underneath you. Without the GIL the code will appear to work, but it will be trivial for an attacker to use mutations to crash the code. Expect huge numbers of CVEs.
This is the lead author of the majorly potential impacted extension!
Borrowed references can be more problematic, but it seems most cases could be fixed by replacing GetItem with FetchItem calls.
Overall, as another C Python extension, I don’t really think it’s going to be that much of a pain. In fact, I could even get away with no changes (other than build fixes and such) if, for example, I guarantee that each “main object instance” is only accessed by one thread, and I’d still get a lot of benefits from the nogil.
if (PyList_Size(lst) >= 3) {
PyList_SetItem(lst, 0, ...);
PyList_SetItem(lst, 1, ...);
PyList_SetItem(lst, 2, ...);
}
if len(lst) >= 3:
lst[0] = "..."
lst[1] = "..."
lst[2] = "..."
IOW, your code is not much different than the equivalent Python code:
if len(lst) >= 3:
lst[0] = "..."
lst[1] = "..."
lst[2] = "..."
References:
• https://docs.python.org/3.8/c-api/list.html#c.PyList_SetItem
This function “steals” a reference to item and discards a reference to an item already in the list at the affected position.
• https://docs.python.org/3/c-api/refcounting.html#c.Py_DECREF
Warning: The deallocation function can cause arbitrary Python code to be invoked (e.g. when a class instance with a __del__() method is deallocated).
• https://docs.python.org/3/extending/newtypes_tutorial.html#a...
But this would be risky. Our type doesn’t restrict the type of the first member, so it could be any kind of object. It could have a destructor that causes code to be executed that tries to access the first member; or that destructor could release the Global interpreter Lock and let arbitrary code run in other threads that accesses and modifies our object.
Let's look at the code:
1. PyList_SetItem calls Py_XSETREF(dst, src) where dst is the list position:
https://github.com/python/cpython/blob/ee46cb6aa959d891b0a48...
2. Py_XSETREF calls Py_XDECREF on the old item:
https://github.com/python/cpython/blob/ca8b55c7f54b38e264056...
3. Py_XDECREF calls Py_DECREF on non-Null objects:
https://github.com/python/cpython/blob/main/Include/object.h...
4. Py_DECREF can cause arbitrary code to run.
I dunno, I'd rather have a reliable bug than a heisenbug.
Any class with a __del__() method is going to release the GIL. That's not uncommon. My intuition is that threaded Python is rarer than classes with a __del__() method.
> Most C API extensions don’t require any changes, and for those that do require changes, the changes are small. For example, for “nogil” Python I’m providing binary wheels for ~35 extensions that are slow or difficult to build from source and only about seven projects required code changes (PyTorch, pybind11, Cython, numpy, scikit-learn, viztracer, pyo3). For four of those projects, the code changes have already been contributed and adopted upstream. For comparison, many of those same projects also frequently require changes to support minor releases of CPython.
If you're using the GIL to protect access to non-Python objects, that will need to change.
The PEP mentions a future HOWTO on how to updating existing extensions. I wish that were already written.
There's disagreement among Python maintainers in the discussion thread on the PEP in how much work will be involved and I don't expect to resolve it here.
This is automatic via the PyList_GetItem/SetItem API, so I guess the error you're talking about is that you read a list y = [A, ...], your code reads A and copies the data to (new) A2, and then you iterate over it again and see that A != A2 because another thread has modified y?
This sounds like a nightmare waiting to happen.
this actually does "break" a lot of things, as you would be surprised how much code implicitly relies upon cPython's behavior of immediately calling weakref callbacks when an object is dereferenced. This is why keeping test suites running on pypy can be difficult, because it has the latter behavior.
> Removing the GIL will require a new ABI, so existing C-API extensions will minimally need to be rebuilt and may also require other changes. Updating C-API extensions is where the majority of work will be if the PEP is accepted.
as you note, there will be *two* versions of the Python interpreter.
that means every C extension has to be built *twice*. against both versions of the interpreter. Go look at how many files one must have available when publishing binary wheels: https://pypi.org/project/SQLAlchemy/#files the number of files for py3.13 now doubles. not clear if we actually have to have both Python builds present, so I would have like /opt/python3.13.0.gil and /opt/python3.13.0.nogil ? if the gil removal changes almost nothing, why have two versions of Python?
This vastly understates the work involved. Most C extensions and embeddings will require major structural changes or even rewrites. These things are everywhere and are a major reason for python's popularity. A typical financial instition, for example, will have a whole bunch of them, with a morass of python code built on top. Many of these companies took ages to transition to python 3 (and many still have pockets that haven't!). For them to remove GIL reliance from their C extensions is a much bigger ask than migrating to 3 was, and their response to being asked to do it is likely to be blunt.
The fact that only a small minority of developers involved in python directly use the C API (and fewer understand the consequences of GIL removal for them) means the issue tends to be overlooked in discussions like this, but it's the reason a GILectomy will be worse than 2 to 3. In practice it's likely to end up being either a fork or a mode you have to switch on/off, both of which would be miserable for everyone.
The GIL is part of python's success story. It made it easy to write extensions, and the extension ecosystem made the language popular. Every lanaguage doesn't have to converge to the same endpoint. Different tools are suited to different jobs. Let it be.
The GIL was one of the reasons why the C module ecosytem took off.
It could be a real headache for any hoped-for transition to nogil Python if lots of GIL-reliant C code is floating around where there's little hope of updating it without having to worry about subtle bugs popping up. And even if the conversion was risk-free (which I doubt), many organizations will still not want to dig into their legacy C codebases and make significant changes.
> In practice it's likely to end up being either a fork or a mode you have to switch on/off, both of which would be miserable for everyone.
Yep. That's the plan.
The misery, you mean?
We python users are more than ready for whatever pain we have to deal with in the future, we are used to misery we live it every day.
First, if the extension isn’t being used in a multithreaded environment, nothing should change. Yes, it isn’t thread safe, but it doesn’t really matter in that context. And given how bad GIL-Python works with threads, I doubt the majority of extensions are written for multithreaded applications.
And the ones that are written for multithreaded are probably already releasing the GIL for long running computations, so they should be written with at least a little bit of thread safety in mind. And in the worst case it shouldn’t be too difficult to hack the code to include a GIL-like lock that needs to be held by library users in order to ensure thread safety without really changing the architecture of the extension.
If an extension doesn't advertise that it is thread safe, the GIL get activated at runtime so it will work exactly like it does now.
For example, if you need to handle the external programs input, stdout and errout, most of the examples you find by googling use threads for this.
A lot of those important "extensions" are actually C and C++ libraries designed for parallelism in their native use which have been made available in Python via bindings (e.g. Pytorch). The cores of these libraries are fine, only the binding layers (often automatically/semi-automatically generated) may need to be updated. I suspect only the C-API extensions designed from scratch to only be extensions are going to have major problems with a no-GIL world.
> Every language doesn't have to converge to the same endpoint. Different tools are suited to different jobs.
People have decided they want to use Python frontends. Plenty of alternative languages could have won out, which would have provided native threading, faster runtimes etc, but they didn't; we are stuck with Python. The continued existence of the GIL is incredibly restrictive on how you can leverage parallelism both in Python itself and in extensions. Only in the simple cases can you make the Python<->extension boundary clean: the second you want to be able to call Python from extension code (e.g. via callbacks or inheritance) the GIL stands in your way.
No, they choose it for the ease of using its performant extensions. And those extensions are fundamentally limited in performance by the existence of the GIL. And the authors of those extensions (and their employers) are behind the work to get rid of it.
There are three groups here:
1. "Pure" Python users, whose code makes little/no use of extensions. GIL removal is an immediate win for these uses.
2. "Good" extension users/authors, whose code will support and benefit from no-GIL. GIL removal is an immediate win for these uses.
3. "Bad" extension users/authors, whose code doesn't/can't support no-GIL. GIL removal probably doesn't break existing code, but makes new uses potentially unsafe.
Maintaining the technical debt of the GIL indefinitely so that group 3 never needs to address its own technical debt is not a good tradeoff for groups 1 and 2 which actively want to move the language forwards.
The above arguments could apply to any breaking changes that a host wants to implement. Those who don’t directly consume the change, those who can adapt, and everyone who is fine with the status quo.
Python is free to do as it pleases, but this breaking change is going to result in a lot of churn.
> GIL removal is an immediate win for these uses.
As a minor aside, the GIL free Python version was actually a performance regression, somewhere between 5-10% impact. https://peps.python.org/pep-0703/#performance . So, an immediate loss for nearly everyone.
But besides being incorrect, the idea that extensions that break on removal of the GIL are "bad" is irrelevant. The question is what the ecosystem will bear. The people who own the codebases I'm talking about are not pushing for GIL removal. They have large, working codebases they're using in production and the cost of redesigning large chunks of them would be real. GILectomy is being pushed by people with varied motives but they certainly don't speak for everyone and their case isn't going to be made successfully by casting aspersions on those with different priorities.
What will happen is that extensions that don't work with the new behavior will be assimilated by the proponents. Support for the new behavior will be hacked in, probably with many mistakes.
The bad results will then be marketed as progress.
keeping a gil when entering and exiting c extensions seems like an obvious steppingstone.
How many people do you think would complain if the python devs. replaced numpy and pythorch with pure pyhton implementations for an April fools joke?
Python the runtime may not be the most performant by itself, but python the ecosystem only took of because of performance hacks like those popular modules, pypi, cython, ... .
> Those modules are largely written in C, not Python.
Yet people choose C to implement those modules because Python is not performant enough and isn't that a nice thought? Lets do something good for humanity and drive another stake into that old abominations heart by making it less relevant.
The issue with python is that the ecosystem is so big, everyone thinks their niche is the one python was "built for". Data science, web backends, sysadmin, app scripting, superclusters, embedding... They all think they're the biggest dog, when the reality is that they're more or less all the same. Pypy and cython are probably popular in some of those niches, but python is not just about them.
The reality is that python "took off" because of two things:
- the syntax
- the ease of interfacing with other worlds, typically (but not limited to) C/C++
Removing the GIL seriously threatens the second reason, because integrations that used to be fairly easy will have to be fundamentally rearchitected. Moving from Python 2 to 3 was trivial in comparison, and it took a decade; this transition might take much longer, or never really happen, with massive losses for the ecosystem. And all for what, some purist crusade that interests only a fraction of the whole community...?
Why?
Once you have a threaded Python, you can allocate one thread to run with a GIL and the other threads without it. Old extensions can access one thread with the old GIL API and new extensions can access all the things with the GIL-less API.
People who want the performance will rewrite their extension. People who don't, won't.
Sadly, you can't just trivially bolt on thread safety to code that was not designed with it in mind.
However, if you go the other way and introduce e.g. an @nogil decorator, similar to that seen in some cpython alternatives, people will have a straightforward path to opt in incrementally as they fix and verify the critical parts of their code, while preserving known working behaviour elsewhere without having to throw entire complex systems over the wall at once.
But, if you want free-threading but you have to use an old extension that is not GIL-less, then you still can, by garding all access to this extension with a lock and setting the PYTHONGIL variable to 0.
But, you will need to compile the extension with --disable-gil or to obtain that from somewhere, if the extension author doesn't provide it, as the ABI of "standard" python and "--disable-gil" python is different.
How --disable-gil builds of python and of extensions will be distributed is not yet totally clear. I don't think pypi has already a clear idea of how to handle those extension, for instance.
Basically there will be two build mode.
The standard build that you get at python.org will work exactly as now (so with a Gil).
The disable-gil build (which will most likely be available through conda or similar) will have the option to run either with the GIL or without. It will automatically detect extension that are not compatible and switch to Gil mode but you can override this with an environment variable.
Could you elaborate on how the Python ecosystem will suffer if these financial institutions get left behind? What sorts of contributions come from these users in particular?
StackOverflow answers, side projects, fanboys
If it doesn't work for you, use the GIL enabled. What's the problem of letting others disable it?
How many people will be surprised / upset that they have to do that once `--disable-gil` is mainlined into the default builds on python.org?
How long before 'build it yourself and use --disable-gil' becomes 'we've enabled --disable-gil by default for Great Victory for Most Users'?
I think it's very very naive to think that the flag will remain an obscure 'use if you want to build python yourself' if the PEP is successful.
> The global interpreter lock will remain the default for CPython builds and python.org downloads.
Is a joke. It means, for now. For the work in this PEP.
I mean, you've literally got the folk from numpy saying:
> Coordinating on APIs and design decisions to control parallelism is still a major amount of work, and one of the harder challenges across the PyData ecosystem. It would have looked a lot different (better, easier) without a GIL.
That isn't 'for a few people who are vaguely interested in obscure technical stuff'; this is the entire python data ecosystem, saying "this will be the default in the future because the GIL is a massive pain in the ass".
> If it doesn't work for you, use the GIL enabled.
Sure, it can be opt-out; but you're fooling yourself if you think this is going to be 'opt-in'.
It will be the default once it's completed, sooner or later, imo.
...and when the 'default' build of python breaks c-extensions, that is a breaking change, even if technically you can build your own copy of python with different compile flags.
brew install [email protected]
apt install python3.12-mp
Two major parallel incompatible implementations of Python, maintained at the same time, duplicating all efforts at testing, dev, build, release train, etc. while both are maintained.
Does it sound familiar?
Even ignoring the obvious parallels to Python 3000, there’s a limited amount of time people will be bothered maintaining both.
Which doesn't mean there will be no GIL. Contrary to popular belief the goal of PEP 703 is not to remove the GIL, but only to be able to disable at runtime if you need free threading, so it is an opt-in feature.
In the interim, it seems that Conda have volonteered to build the extensions that are frequently used in the scientific community.
Sam claims otherwise: "Most C API extensions don’t require any changes, and for those that do require changes, the changes are small. For example, for “nogil” Python I’m providing binary wheels for ~35 extensions that are slow or difficult to build from source and only about seven projects required code changes (PyTorch, pybind11, Cython, numpy, scikit-learn, viztracer, pyo3). For four of those projects, the code changes have already been contributed and adopted upstream. For comparison, many of those same projects also frequently require changes to support minor releases of CPython."
https://discuss.python.org/t/pep-703-making-the-global-inter...
In the following talk, he shows the scikit-learn changes that were needed:
https://youtu.be/9OOJcTp8dqE?t=1958
I'd also like to point out that the GIL does not inherently make C extensions thread safe. It only protects the interpreter itself. For example, calling in to the C API can already release the GIL as I explained in reply to /u/bjourne here:
Python is a very slow language whose main advantage is relatively cheap and easy FFI with bindings to all imaginable library already written.
With breaking FFI you'll lose all the libraries while adding SMP to still slow language. It's a cleare loss of value.