My understanding is the GIL does not protect against Python-side bugs, and bugs from GIL removal would only be introduced from C extensions.
This has been discussed extensively in the past (1), and my understanding of the take away was that the GIL doesn't protect you from arbitrary execution order; it protects you from undefined behavior due to concurrent write/read in parallel scopes and the resulting data corruption.
...which, as I understand it, there is no specific reason it would be restricted to native extensions.
Is there some more detail to the nogil proposal that addresses the type of UB you see in eg. c, with this? (Wouldn't that require that at some level the GIL still exists?)
Not really, no. The finer grained synchronization primitives are (a) already available in Python, and (b) necessary even with the GIL for reasons I've given elsewhere in this discussion.
What nogil does is enable multiple threads to run Python bytecode at the same time, so that CPU intensive operations in Python can be parallelized without having to use multiple processes. Python objects that are accessed from multiple threads will have to be guarded with synchronization primitives under some circumstances where, in principle, they don't need to be guarded now (operations that only take a single bytecode), but in practice I don't think that will make much difference. The big issue, as has been mentioned elsewhere in this discussion, is C extension modules.
The finer grained synchronization primitives does not already available in Python. Or, it should not be visible to Python code at all. What I'm talking about is the internal implementation of, e.g. PyDict. While from Python bytecode side setitem on it is already not atomic, it does guarantee that Python interpreter won't segfault if there are two Python threads manipulating one dict object concurrently. This is achieved via GIL and has to be replaced.
It's the same problem you mentioned above as "problems in C extension". But no, nogil is hard not only because of compatibility issues. People (especially those who insist on that their workload is inherently embarrassingly parallel) do NOT accept any regression in single-thread performance. If you only ever want to optimize for single thread one global lock is the optimal solution.
No, it doesn't, except for operations that complete in a single byte code. See my response to shrimpx downthread.
like l=[], then l.append(1) and l.append(2) running concurrently will not end up in some weird scenario where the length of the list is 1 yet you stored two items... anyways, I agree with the comment you posted higher up in the discussion, and that was my understanding.
Yes, and I weighed in on similar lines in that discussion:
a += b
takes four bytecodes: two LOAD bytecodes to put the values of a and b on the stack, an INPLACE_ADD bytecode to put the result at the top of the stack, and a STORE bytecode to store the result in the variable a. A thread switch could take place in between any two of these bytecodes, and if another thread mutated a or b or was reading the value of a or b, you would have data corruption. The GIL does not prevent this. The only way to prevent it would be to use one of the locking mechanisms provided to guard access to the a and b variables so that only one thread could access them at a time.
import threading
i = 0
def test():
global i
for x in range(1000000):
i += 1
threads = [threading.Thread(target=test) for t in range(20)]
for t in threads:
t.start()
for t in threads:
t.join()
assert i == 20000000, i