Why is this different from Java, where the JVM can assume that local variables cannot be changed by other threads?
That is, in the absence of special cases, which I believe include `volatile` and function calls. (Any other special cases?)
These are distinct from eg Spectre because the point isn't isolation breaking, it's leakage. But at least a few years ago there was always a question of "are side channel attacks real", and I think we're pretty clearly over that question now.
You can easily pass the address of a thread_local variable around. Threads can easily muck around with objects from multiple threads. Hell that's even true with Java too.
The reason threading is weird on Python is due to the GIL. Optimizing for threads is tough because Python explicitly avoids defining a threading model & assumes all implementations have something like the GIL. That doesn't mean no optimization is possible.
- Don't check. Programs will crash if a thread-local variable is passed outside the thread (C, C++)
- Naive interpreter. Everything works, slowly, because the worst case code is used for everything. (CPython)
- Really clever just-in-time recompilation when what seemed to be thread-local suddenly gets accessed from another thread. (PyPy, Java?)
- Compile time checking to prevent this. (Rust)
- No threads (classic Javascript)
- Explicit shared areas into which only certain types can be placed (ECMAscript 2018)
1. Python doesn't have thread-local variables 2. Python's memory model around threading is what prevents any optimization.
Both are clearly incorrect. For example, take Java. You have the same behaviour as Python: your VM won't crash & your code won't even generate a panic. Java clearly has thread-local variables & an optimizing compiler.
Python's inhibition around optimizing is partially around the guarantees they make around APIs that do self introspection but mainly around keeping the interpreter simple. For a counterpoint see PyPy, Cython, JPython, IronPython etc which optimize Python just fine. They mostly trip over module extensions being tied directly to the non-standardized CPython module API & that API is tied intimately to implementation decisions of CPython.
The default python interpreter does absolutely nothing to make it's execution thread-safe, and that appears to be a deliberate decision on their part to keep the complexity of the interpreter down.
For example, a single list can safely be extended (as in `l += [1]`) concurrently from several threads.
The GIL just means that only one of those threads will be executing pure-Python code at any time. But they're still normal threads.
It's tough to optimize a language where any thread can change the internal variables of any other thread at any time. Python has gratuitous dynamism. Rarely does code muck with the state of another thread, but it can, and the code has to handle the worst case. There's no such thing as a thread-local variable in Python.