News for hackers. Stuff that matters.
Re:Python 3 in 2016 by ivoras on Thursday February 18, 2016 @05:19AM (#11124798)
Since it's basically all anecdotal, I'll say my gazing-into-the-crystal-ball educated-guess feeling that Python 4 will make the entirety of Python about as popular as Perl is today if: it doesn't get rid of GIL and at the same time implement a JIT compiler, and at the same time remain 100% .py source-code compatible with Python 3.

The developers have dug themselves into this mess with frivolous differences ("print" a function, wanting to remove the "%" string operator, etc.), instead of spending the years on more productive features, now is the chance to dig themselves out again.

It's a beautiful language, too bad the development choices were a bit sketchy.

Re:Python 3 in 2016 by sametmax on Thursday February 18, 2016 @08:11AM (#11125450)
Python 3 is much more than changing the print() statement (which, actually, is a great thing). It's about unicode by default. It's about removing 100 of duplicates way to do things to have only one. It's about making debug much, much easier, while at the same time allowing you to write better code with less mistakes. It's about removing a lot of boilerplate. It's about introducing new syntaxes allowing things not possible in python 2.

If you see only print, you are missing the point.

But good news, there are many projects working to improve the perfs and/or bypass the GIL this year.

Re:Python 3 in 2016 by plesner on Thursday February 18, 2016 @10:03AM (#11126190)
> If you see only print, you are missing the point.

The top thing, sometimes the only thing, that makes my scripts written for 2 not work in 3 is the change to print. Adding a print function is fine, I'm all on board with that; removing the statement though seems like pain for no reason. Bringing that up is not missing the point, it's hitting the nail right on the head.

Re:Python 3 in 2016 by JupiterMoon on Thursday February 18, 2016 @12:25PM (#11127477)
I think you might want to look into the 2to3 script. It should handle the conversion of your print statements.
Re:Python 3 in 2016 by ZenoArrow on Thursday February 18, 2016 @05:33AM (#11124849)
> "it doesn't get rid of GIL and at the same time implement a JIT compiler, and at the same time remain 100% .py source-code compatible with Python 3."

Guido could do that if he decided that Python 4 would be based on PyPy (AFAIK, many people thought Python 3 would be based on PyPy too). Compatibility isn't 100% but if it was the main platform I'm sure it'd improve even faster.

Re:Python 3 in 2016 by dagw on Thursday February 18, 2016 @06:14AM (#11125013)
There is also pyston, which is a 'competing' LLVM backed JIT that Dropbox is working on. Guido also happens to be working at Dropbox, but I don't know if he's directly involved with pyston or not.

AFAIK, many people thought Python 3 would be based on PyPy too

Don't think so. Some people hoped that Python 3 might be based on Unladen Swallow, which was a different Python JIT that Guido was working on while at Google, but that project never really got off the ground and Google seemed to lose interest in it.

Re:Python 3 in 2016 by chucksmash on Thursday February 18, 2016 @09:17AM (#11125846)
>Guido also happens to be working at Dropbox, but I don't know if he's directly involved with pyston or not.

With the proviso that a lot can change in two years, here's a relevant comment from the Dropbox Pyston announcement [1]:

"Guido's advice has been extremely helpful, but so far we haven't been able to get any code from him"

[1]: https://blogs.dropbox.com/tech/2014/04/introducing-pyston-an...

Re:Python 3 in 2016 by throw12873 on Thursday February 18, 2016 @06:25AM (#11125048)
Why do people always give a single person all the credit? I don't think Guido was ever really involved in Unladen Swallow or that he has any particular interest in JIT compilers.
Re:Python 3 in 2016 by dagw on Thursday February 18, 2016 @06:27AM (#11125054)
I don't think I gave him all the credit. We where talking about Guido and I mentioned a relevant project I thought (perhaps erroneously) that used to work on.
Re:Python 3 in 2016 by pyvpx on Thursday February 18, 2016 @06:10AM (#11125006)
pypy still has the GIL.
Re:Python 3 in 2016 by Spiritus on Thursday February 18, 2016 @06:58AM (#11125140)
Re:Python 3 in 2016 by DasIch on Thursday February 18, 2016 @08:51AM (#11125684)
STM isn't magic. Transactions can and will silently fail and you lose your parallelism. You need additional tools to identify and debug failed transactions and you'll need to learn how to structure your code to avoid failed transactions.

This might turn out to be quite difficult for non-trivial Python applications.

STM is exciting but I think it's important to keep in mind that it's an experiment that while promising might still fail in practice.

Re:Python 3 in 2016 by ZenoArrow on Thursday February 18, 2016 @10:13AM (#11126276)
> "STM isn't magic."

What difference does that make. STM will still enable the removal of the GIL. If an applications is coded that doesn't work with PyPy, that's a separate matter.

Re:Python 3 in 2016 by jerf on Thursday February 18, 2016 @11:35AM (#11126944)
But first it has to actually work. So far, based on that page, it has suffered the same fate as all the other attempts to remove the GIL; mostly functional, but slower. And the numbers cited on that page put it on the rather slow side for single-threaded code for a GIL removal, too, which is what has always prevented previous efforts from getting merged in.

I remain skeptical that STM is a good fit for this problem, as I was when I first heard about this, and after their work, alas, I have to say I see little evidence to make me change my mind on that. STM is something that probably needs to be baked in from day one, possibly all the way into the language itself, not put on the side of a complicated secondary implementation of Python.

Re:Python 3 in 2016 by ZenoArrow on Thursday February 18, 2016 @11:51AM (#11127088)
> "So far, based on that page..."

From the page... "THIS PAGE IS OLD, THE REST IS ABOUT STMGC-C7 WHEREAS THE CURRENT DEVELOPMENT WORK IS DONE ON STMGC-C8"

This page is probably a better starting point for the current PyPy STM work:

http://pypy.org/tmdonate2.html

Re:Python 3 in 2016 by jerf on Thursday February 18, 2016 @02:03PM (#11128318)
I didn't see anything newer there to suggest they've made a lot of progress since that page, though.

Part of the problem is that they're trying to wedge into a really tight window; for all its flaws, the GIL is not all that expensive (relative to how slow Python in general is, anyhow), and trying to do it on PyPy makes their window even smaller. It's a really hard thing to hit, and I'll believe it when I see it. I'm not saying it's impossible. Just, I'll believe it when I see it, and I don't recommend a high degree of confidence that it will occur. Nor do I recommend that you use it as part of your advocacy... it's nowhere near solid enough to be making promises about it to people outside the community.

Re:Python 3 in 2016 by ZenoArrow on Thursday February 18, 2016 @03:41PM (#11129134)
> "I'll believe it when I see it".

You may be interested in this...

"Leysin Winter Sprint (20-27th February 2016)"

"STM (Software Transaction Memory), notably: try to come up with benchmarks, and measure them carefully in order to test and improve the conflict reporting tools, and more generally to figure out how practical it is in large projects to avoid conflicts"

http://morepypy.blogspot.co.uk/2016/01/leysin-winter-sprint-...

> "Nor do I recommend that you use it as part of your advocacy."

I'm an advocate for F#, but I'll encourage the progress of any open source language, strong competition is good for the field as a whole. The question isn't 'Why can't Python be fast?' but rather 'How fast can Python be?', the difference is a difference in attitude.

Re:Python 3 in 2016 by orf on Thursday February 18, 2016 @05:28AM (#11124826)
Comments like this are so misguided and yet so common on HN. Adding a freaking JIT to CPython is a ridiculous suggestion that will never happen for a whole load of reasons. Getting rid of the GIL is more noble, but seems impossible to do without breaking compatibility.

Python 4 will just be the next release after Python 3.9.

Re:Python 3 in 2016 by dagw on Thursday February 18, 2016 @06:04AM (#11124987)
Getting rid of the GIL is more noble, but seems impossible to do without breaking compatibility.

The problem isn't just breaking compatibility, it's performance. There have been a number of experiments with removing the GIL going back (I think) to python 1.4. However every attempt has led to significant performance degradation in some part of the language.

Re:Python 3 in 2016 by plesner on Thursday February 18, 2016 @09:06AM (#11125769)
> Adding a freaking JIT to CPython is a ridiculous suggestion that will never happen for a whole load of reasons.

That's a bold statement. Can you elaborate on what those reasons are?

Re:Python 3 in 2016 by orf on Thursday February 18, 2016 @09:58AM (#11126142)
It's not a bold statement, it's one made time and time again by the cPython developers. In short:

1. The Python language itself doesn't lend itself to being JIT compiled - it's too dynamic.

2. cPython is the reference implementation. It's source should be as simple as possible.

3. Only a subset of Python programs would benefit, and the rest would be slower/more memory hungry

and finally,

4. If you want/need a JIT then use PyPy, in all it's glory with all its upsides and downsides.

PyPy is a big project with a lot of very very very smart people working on it. They've done an excellent job at JITting Python (and creating an impressive interpreter framework) but even they couldn't make a JIT that is anywhere near suitable for inclusion in cPython. A bunch of cPython hackers on python-dev have no chance.

tldr: Want a JIT? Use PyPy.

Re:Python 3 in 2016 by plesner on Thursday February 18, 2016 @10:22AM (#11126349)
By "bold" I'm also referring to your use of the words "freaking" and "ridiculous".

1. This was also said about JS before people just went ahead and wrote JITs for it. Are you sure what they're saying isn't "we can't make a JIT for python" rather than "nobody can make one"? Because that's how it turned out with JS.

2. If given the choice, do you think the community would turn down a JIT to avoid making the reference implementation more complex? To whose benefit is that requirement? I've never heard people complain about other languages that they didn't have an easy-to-read reference implementation.

3. I've not heard this reported as a problem in practice for languages that have a JIT. It's usually toy programs where the JIT overhead is significant and they're so small and short-running that neither slowdown or extra memory is usually an issue.

4. Not everything that runs on cPython runs on PyPy so that's not a choice I'm necessarily able to make.

I'm not saying there aren't reasons to not have a JIT in cPython. But it's a choice not a natural law. And it's not ridiculous to suggest that they could make a different choice if they wanted, as JS eventually did.

Re:Python 3 in 2016 by jerf on Thursday February 18, 2016 @11:29AM (#11126886)
JITs for Javascript aren't comparable; at the time, nobody had C-based extensions for Javascript. It makes it a radically simpler problem.

Remove the "maintaining compatibility for C-based extensions" requirement, and you get, well, PyPy. And you also get a community split that makes 2/3 look trivial. Vast swathes of core Python code and libraries are really not "Python", but Python C-based bindings to custom code or C libraries.

Re:Python 3 in 2016 by plesner on Thursday February 18, 2016 @12:00PM (#11127186)
Sure there were C-based extensions for JavaScript, that's how it integrates with the DOM. It makes it a little trickier but not a whole lot.
Re:Python 3 in 2016 by jerf on Thursday February 18, 2016 @01:54PM (#11128239)
Sorry, perhaps I was unclear. I mean, there were no C-based extensions not under the control of the same project as controlled the JS engine, so one entity can both JIT a JS engine and simultaneously make whatever modifications it took to make them work under the new engine. Python has an enormous amount of external C-based extensions not controlled by any one entity, which constitute huge swathes of core (in importance, not in "under control of the core team") Python libraries. It's difficult to imagine what a JIT looks like that still works with those. You don't have one entity doing both the JIT and all the C extensions.
Re:Python 3 in 2016 by orf on Thursday February 18, 2016 @11:01AM (#11126654)
1. By people you mean "Google", and with a whole lot of money and a room full of some of the smartest people they could find. V8 is a whole department working full time, not a open source community-led effort. JS is also a different beast in a whole lot of ways. No, I am definitely saying "we can't make a JIT for Python that doesn't significantly impact single threaded scripts, retains significant backwards compatibility and doesn't massively complicate the cPython code". Obviously someone can make one, PyPy did, and it would be insane to ignore all of the HUGE problems they encountered (and overcame) while writing a JIT for Python. What makes you think cPython could do better?

2. It would make the reference implementation insanely more complex, to the point where it is no longer a reference implementation but a JIT implementation of Python. That's not cPython's role. For whos benefit is the requirement of a JIT? My apps run fine without them, and I don't need the memory bloat or startup cost associated with a JIT. If the community really wants another JIT they could fork cPython, find a room full of JIT experts and re-do all of the work PyPy has done, with all of the tradeoffs and problems they encountered. Nice idea.

3. "so small and short-running that neither slowdown or extra memory is usually an issue." - this is exactly the situation where a JIT adds a lot of overhead and becomes an issue.

4. So you want the best of both worlds, with no clear idea as to what magical person can make this happen (or even how) and you won't be happy until you can have your cake and eat it?

Re:Python 3 in 2016 by plesner on Thursday February 18, 2016 @12:44PM (#11127625)
You're making it sound like you're disagreeing with me but it sounds like we're in complete agreement on the main point: it's not impossible to create a cPython JIT. It's a choice whether to do it or not. (And by the way, creating a JIT is a piece of engineering, it's not black magic). It's fair enough if people think that would be the wrong choice, I might even agree actually, but if the cPython developers say, time and time again, that python is inherently unsuitable to be JITted then that's wrong and it prevents that discussion from taking place.
Re:Python 3 in 2016 by orf on Friday February 19, 2016 @04:58AM (#11132624)
Few things are impossible to code. It's hard to create a JIT for Python and the reference implementation for Python is not the right place to add such a JIT for the reasons stated in the comments above.

It's not that the Python developers preventing discussion, it's just this discussion is always fruitless, has been had many times before and the people invoking the discussion often blame the cPython developers for their response. Nobody wins, it's a waste of time.

Re:Python 3 in 2016 by knabacks on Thursday February 18, 2016 @11:42AM (#11127005)
"tldr: Want a JIT? Use PyPy."

or spend some time helping with Pyjion to add a C API to CPython for plugging in a JIT of your choice

- https://github.com/Microsoft/Pyjion

Re:Python 3 in 2016 by bhaak on Thursday February 18, 2016 @05:47AM (#11124912)
> Getting rid of the GIL is more noble, but seems impossible to do without breaking compatibility. > > Python 4 will just be the next release after Python 3.9.

Really? Why even change the major version then, when there are no breaking changes?

Given that many Python programmers seem to be traumatized by the last major version switch, why even bother?

Or is this just for setting up Python 5 can have breaking changes if the switch to 4 is painless?

Re:Python 3 in 2016 by orf on Thursday February 18, 2016 @06:54AM (#11125130)
> Really? Why even change the major version then, when there are no breaking changes?

Python releases don't follow semver. If Linus can bump the kernel to 4.0 because he felt like it so can Python.

Re:Python 3 in 2016 by dragonwriter on Thursday February 18, 2016 @09:27AM (#11125914)
> If Linus can bump the kernel to 4.0 because he felt like it so can Python.

"Can" is not "should": I mean, Python can go from 3.x to Python 2020, go with year numbers for a while, then have Python XP, then go Python 7, Python 8, Python 8.1, and then skip to Python 10. But there's no good reason to do that.

Re:Python 3 in 2016 by ivoras on Thursday February 18, 2016 @05:47AM (#11124911)
It may very well never happen, that's a completely possible outcome.

We'll see what it does to Python's popularity.