2. There are a lot of users that want fast Python and a lot of effort was put in to optimise it. I think the core team wouldn't want to release and expect everybody to use a version which regresses in performance, for a feature (nogil) which most won't be able to use due to using C extensions which haven't had nogil-supporting code changes.
It looks for a flag, if it doesn't see it then it turns the GIL on.
Where is the need for recompilation?
> I think the core team wouldn't want to release and expect everybody to use a version which regresses in performance
But you're just guessing that it's slower in GIL mode, aren't you?
https://docs.python.org/3/c-api/typeobj.html#c.PyObject.ob_r...
The stable ABI lets extensions access the reference count of any object directly. I don't know why. Normally the functions Py_IncRef and Py_DecRef should be sufficient. Objects no longer have a single number as their reference count.
Edit: In Python 3.2-3.9, the stable ABI included Py_INCREF, the C macro.
https://docs.python.org/3/c-api/type.html#c.PyType_FromModul...
The stable ABI lets extensions create new Python types. These can override tp_alloc field to use custom memory allocators when instances of the type are instantiated. The custom memory allocator needs to initialise the reference count to 1.
> But you're just guessing that it's slower in GIL mode, aren't you?
There are new implementations with per-object locks of list.append, dict.__setitem__ etc. These are incredibly common operations in Python code. These inherently will be more complicated than the previous implementation, meaning more instructions and slower. So there needs to be a branch at the start of list.append of whether to go to the old gil implementation or the new nogil one. Adding a branch so frequently will inherently make the runtime slower.
Now with a lot of work these can be optimised and the speed penalty reduced, but CPython goes on an annual release cycle. If the nogil code is held as a fork of the CPython repository without being merged in, until it reaches a performance goal, that brings other issues.
Does that stay true once the GIL turns back on?
> These can override tp_alloc field to use custom memory allocators when instances of the type are instantiated. The custom memory allocator needs to initialise the reference count to 1.
I'm not following why this affects ABI compatibility, sorry.
> So there needs to be a branch at the start of list.append of whether to go to the old gil implementation or the new nogil one. Adding a branch so frequently will inherently make the runtime slower.
I don't know about that. CPUs handle always-taken and never-taken branches very well.
In the current nogil implementation, AFAICS, it seems the GIL can't be turned back on so there is no answer yet.
Theoretically, you could have a one-off operation which fixes all objects when the GIL is turned on. However, there's no way to get all objects in Python. gc.get_objects() only returns tracked objects, and there is no way to list untracked objects.
It seems, three fields are exposed on every CPython object in the stable ABI, any change which affects their offsets will break the stable ABI. https://github.com/capi-workgroup/problems/issues/4#issuecom...
> I'm not following why this affects ABI compatibility, sorry.
True, PyType_FromSpec can set tp_alloc to a wrapper function which papers over the difference in what the "allocfunc" should initialise the memory to.
Re performance - merging the change is the only way people will actually start targeting nogil.
and dynamic patching is also an option.
In either case it seems obvious now that there is likely no point bothering with the stable ABI any more.
Yes, that's what they were asking about. Why have two versions for a mode switch. Everything you explained before that is irrelevant, I'm afraid.
> but I'm expecting this to be slower than running the gil build in GIL mode.
That could be the answer to their question, but that's not definitive enough.