For PyParallel I just altered the decl for global variables to be TLS static, then tweaked initialization, and voila, thread safe globals.

E.g. https://bitbucket.org/tpn/pyparallel/src/3be2954508f9938b85a...

The problem is that thread local storage is slow. It's an extra layer of indirection. Also, thread safety is just one concern in the GIL debate. One should be able to support many low-overhead independent interpreter instances on a single thread rather than relying on an arcane interpreter context switching scheme as presently exists in Python 2 and 3.

> The problem is that thread local storage is slow.

It's slower than accessing a struct, sure:

    5982:     if (ctx) {
    000000001E1A808B 8B 0D FF 7C 28 00          mov         ecx,dword ptr [_tls_index (1E42FD90h)]  
    000000001E1A8091 BA 80 29 00 00             mov         edx,2980h  
    000000001E1A8096 48 89 83 90 02 00 00       mov         qword ptr [rbx+290h],rax  
    000000001E1A809D 65 48 8B 04 25 58 00 00 00 mov         rax,qword ptr gs:[58h]  
    000000001E1A80A6 48 8B 04 C8                mov         rax,qword ptr [rax+rcx*8]  
    000000001E1A80AA 48 8B 0C 02                mov         rcx,qword ptr [rdx+rax]  
    000000001E1A80AE 48 85 C9                   test        rcx,rcx  
    000000001E1A80B1 74 70                      je          new_context+253h (1E1A8123h)

But think of the big picture: I use TLS everywhere for PyParallel, and PyParallel has awesome performance, so eh, net win.

> One should be able to support many low-overhead independent interpreter instances on a single thread

Why? What problem does that solve?

One use case, totally independent of parallelism is that if someone say embeds Python inside of Postgres or some other large system, I cannot also, in my extension, also embed Python. First one wins and now our systems have to agree. In Lua, you allocate an interpreter context and use that. There can be any number of Lua interpreters embedded in a large system without conflicting.

The use of the globals in the CPython interpreter are a fairly large design mistake that has prevented Python from having as much reach as other systems. The whole embedding vs extending debate is because of those globals and that CPython has been historically difficult to embed properly. Lua on the other hand, is easy to both embed and extend.

I'd love to use `cffi` to embed Python within itself, I cannot do that.

A one time cost of 100 hours of work to put the globals into an interpreter struct and fix the C API to have a PythonInterpreter* as the first argument to every function would be a very small price to pay compared to the decade-long debates and hacky workarounds surrounding the bad GIL design.

One side effect of Nick Coghlan's PEP 432 (interpreter startup) is that is makes the effort of consolidating global state (out of static variables) easier. His implementation is progressing. My proposed subinterpreter-based project will likely involve the effort of pulling all remaining global interpreter state into the interpreter struct.

As to supporting multiple truly independent Python interpreters in the same process, I'm not clear on what the buys you over subinterpreters. In fact, with subinterpreters you get the benefit of the main interpreter handling the C runtime global state (env vars, command line args, the standard streams, etc.). With truly independent interpreters that's more complicated.

While it's not quite what you described, if you were to suggest that subinterpreters be made even more isolated from one another than they already are then I'd agree. :) It sounds like that's really what you're after: better isolation between interpreters in a single process.

Maybe now that Python 2.7 dev is slowing down the change could be done. But historically, good ideas don't necessarily get adopted in CPython. Stackless was never allowed in, and I'd say it is pretty compelling fork, and actually one that would help with this exact problem. So, unless the change was blessed from before the work commenced, I am not