News for hackers. Stuff that matters.
Re:Tornado Without a GIL on PyPy STM by Animats on Monday November 17, 2014 @02:28PM (#8619942)
Python's Little Tin God wouldn't like it.

Python's feature set is basically what's easy to do in a naive interpreter. Everything is a dictionary. Anything can be changed from anywhere. With "getattr", you can patch one thread from another. It's elegant, and very difficult to speed up. Google tried, with von Rossum on board. Their "Unladen Swallow" JIT compiler project crashed and burned.

The PyPy group has made a fast Python compiler/interpreter/JIT system. It's really hard. The initial funding from the European Union got them started, but wasn't enough. They really try to handle all the hard cases. This requires two interpreters and a JIT. They have to handle "oh no, someone patched object A from thread R, invalidating code that's running in thread S". There's a "backup interpreter" that kicks in for hard cases, and once control is out of the area in trouble, the JIT can recompile it. (This is an old and oversimplified description.)

This transactional memory thing is very clever. It has to separate things at run time that probably should have been separated at compile time, of course. It's impressive that they can get it to work. It's a lot like how a superscalar CPU works, including transaction commit and backup at the retirement unit.

Python gets into this mess because, like C and C++, the language doesn't really know about concurrency. (Threads came late to UNIX, and C predates threads. So C has an excuse for backing into concurrency.) Python has the C model of concurrency; at the user level, it's treated as a library issue. Internally, though, it needs a lot of locking, because there's so much mutable state in the interpreter.

It would be a lot easier if the language were restricted a little. But then It Wouldn't Be Python(tm). The price of this is huge complexity layered on a simple model, and probably years of obscure bugs in PyPy.

Re:Tornado Without a GIL on PyPy STM by beagle3 on Tuesday November 18, 2014 @06:55AM (#8623507)
> Python has the C model of concurrency; at the user level, it's treated as a library issue.

Other than Erlang, which modern language doesn't?

Java has e.g. "synchronized" keyword, but I regard it as merely syntactic sugar over a standard library implementing thread - semantically, it still basically has the C model.

Go has channels and stuff - but it's still library level (in fact, it's basically syntactic sugar over the Plan9/Inferno/Aleph/DontRememberName standard library primitives)

Re:Tornado Without a GIL on PyPy STM by pjmlp on Monday November 17, 2014 @04:21PM (#8620672)
> C predates threads

There were other OS with threads and co-routines being explored as design while UNIX was being developed at AT&T.

As for the rest I agree with you.

Personally I don't have any use for Python besides the occasional shell script, but as a user of applications written in Python I would like them to perform fast.

Re:Tornado Without a GIL on PyPy STM by Animats on Monday November 17, 2014 @04:27PM (#8620718)
Yes, and UNIVAC 1108 Exec 8 had threads in 1967. But it wasn't on UNIX until the 1980s.
Re:Tornado Without a GIL on PyPy STM by fulafel on Tuesday November 18, 2014 @04:01PM (#8626327)
Local variables arent dict based, and attributes are only sometimes (slots are the other option).

Unladen swallow was attempted at a time when llvm jit was buggy and slow.