Re:Meta pledges Three-Year sponsorship for Python if GIL rem...
by js2 on Sunday July 09, 2023 @06:46PM
(#36659814)
> Most C extensions and embeddings will require major structural changes or even rewrites.
Sam claims otherwise: "Most C API extensions don’t require any changes, and for those that do require changes, the changes are small. For example, for “nogil” Python I’m providing binary wheels for ~35 extensions that are slow or difficult to build from source and only about seven projects required code changes (PyTorch, pybind11, Cython, numpy, scikit-learn, viztracer, pyo3). For four of those projects, the code changes have already been contributed and adopted upstream. For comparison, many of those same projects also frequently require changes to support minor releases of CPython."
I'd also like to point out that the GIL does not inherently make C extensions thread safe. It only protects the interpreter itself. For example, calling in to the C API can already release the GIL as I explained in reply to /u/bjourne here:
Sam claims otherwise: "Most C API extensions don’t require any changes, and for those that do require changes, the changes are small. For example, for “nogil” Python I’m providing binary wheels for ~35 extensions that are slow or difficult to build from source and only about seven projects required code changes (PyTorch, pybind11, Cython, numpy, scikit-learn, viztracer, pyo3). For four of those projects, the code changes have already been contributed and adopted upstream. For comparison, many of those same projects also frequently require changes to support minor releases of CPython."
https://discuss.python.org/t/pep-703-making-the-global-inter...
In the following talk, he shows the scikit-learn changes that were needed:
https://youtu.be/9OOJcTp8dqE?t=1958
I'd also like to point out that the GIL does not inherently make C extensions thread safe. It only protects the interpreter itself. For example, calling in to the C API can already release the GIL as I explained in reply to /u/bjourne here:
https://news.ycombinator.com/item?id=36649769