News for hackers. Stuff that matters.
Re:PyPy2.7 and PyPy3.5 v5.7 released by bsaul on Tuesday March 21, 2017 @07:30AM (#13921272)
This project keeps amazing me. It seems like a R&D lab going for moonshots on the python language, and yet achieving incredible practical results.

Does anyone know why it still hasn't managed to get enough traction to become the official path for the language, and why it still needs mozilla support to get funds ? My intuition is that it's just too complex. But haskell,scala,swift and rust also have a lot of complexity so that can't be the only reason.

Re:PyPy2.7 and PyPy3.5 v5.7 released by wyldfire on Tuesday March 21, 2017 @09:17AM (#13921878)
> Does anyone know why it still hasn't managed to get enough traction to become the official path for the language

I'd wager BDFL prefers the CPython implementation because it's probably simpler. Also, it supports tons of targets and is super simple to build.

It's healthy for a language to have more than one implementation. A big win would be a linux distro selecting pypy for use as the default Python. But IMO many python scripts that folks use just don't run long enough to see a benefit from PyPy.

Re:PyPy2.7 and PyPy3.5 v5.7 released by ronjouch on Tuesday March 21, 2017 @09:52AM (#13922170)
> "I'd wager BDFL prefers the CPython implementation because it's probably simpler. Also, it supports tons of targets and is super simple to build."

You're doing good wagering, Guido mentioned just this at several keynotes (source: skim through his PyCon/EuroPython keynotes, there's a good chance he addresses the point or answers a question about it). I'd paraphrase him with: although now a big project with optimizations here and there obscuring the purpose of many parts, CPython remains a boring well-understood "no rocket science" C project. As an example of this, I remember him mentioning how the opcode dispatcher remains a giant `switch` statement rather than something more intricate.

That doesn't mean he dislikes or wants to hinder PyPy (or IronPython or Jython or Pyston or ...), just that he's fine with CPython reach and trade-offs as the "default python", and if you prefer another Python with different performance/compatibility/featureset/xyz characteristics, you're welcome to grab it :)

Re:PyPy2.7 and PyPy3.5 v5.7 released by kzrdude on Tuesday March 21, 2017 @01:30PM (#13924341)
But it sounds like he implicitly hinders CPython! It sounds like “we won't merge code I don't understand” and no brave intelligent soul will risk improving the interpreter.
Re:PyPy2.7 and PyPy3.5 v5.7 released by nameless912 on Tuesday March 21, 2017 @11:56PM (#13928349)
Isn't that the explicit job of a BDFL though? To enforce standards? His happens to be simplicity at all costs, and while some might disagree, I think it's generally a good pursuit; as others state, alternative implementations are available if you really need some JITing black magic or Java integration or green threads or...
Re:PyPy2.7 and PyPy3.5 v5.7 released by milesrout on Wednesday March 22, 2017 @02:13AM (#13928818)
What's wrong with "we won't merge code we don't understand"? I think a lot of people should hold code to that standard. I think almost everyone does hold code to that standard.
Re:PyPy2.7 and PyPy3.5 v5.7 released by kzrdude on Wednesday March 22, 2017 @05:27AM (#13929539)
It's a problem when leadership is not ~~merit~~ engagement based (BFDL) and when said leadership is lacking the vision and gut to push Python across new boundaries. I would like to see something more “aggressive” here.
Re:PyPy2.7 and PyPy3.5 v5.7 released by milesrout on Wednesday March 22, 2017 @06:38PM (#13935689)
The leadership of Python, in the form of Guido, is certainly merit- and engagement-based. Nothing needs to be 'more aggressive' in Python development. It's perfectly fine for the standard, canonical version of Python to be the version that:

* Is most up-to-date and has all the latest features * Everyone is most familiar with * Already is packaged in all operating systems * Is the most stable * Is compatible with all C extensions

Re:PyPy2.7 and PyPy3.5 v5.7 released by data_hope on Tuesday March 21, 2017 @09:30AM (#13921972)
> I'd wager BDFL prefers the CPython implementation because it's probably simpler. Also, it supports tons of targets and is super simple to build.

At a EuroPython Keynote, the BDFL mentioned that he hasn't had a closer look at PyPy (he mentioned downloading it and playing with it for a few minutes). I.e. there is a certain disinterest. Also, remember that the "Zen of Python" (https://www.python.org/dev/peps/pep-0020/#id3) was written about the design principles of the Python interpreter, and PyPy is not exactly the Zen of python.

Personally, I'd love to see Python 4 to be based entirely on PyPy.

Re:PyPy2.7 and PyPy3.5 v5.7 released by catdog on Tuesday March 21, 2017 @09:29AM (#13921963)
> Does anyone know why it still hasn't managed to get enough traction to become the official path for the language

One reason is that CPython is kept relatively simple on purpose, so it abstains from implementing things like a jit or other very complex optimizations. The biggest obstacle for pypy adoption is that it can't serve as a drop in replacement in the majority of cases, be it because its behind (until now far behind) in supporting new versions of the language or because a lot of popular packages rely on C extensions. The CPython way of binding to C on which these packages rely on does not fit the architecture of pypy very well so they did not really support it at all until recently.

Re:PyPy2.7 and PyPy3.5 v5.7 released by hyperbovine on Tuesday March 21, 2017 @01:02PM (#13924062)
> Does anyone know why it still hasn't managed to get enough traction to become the official path for the language

The people who really care about Python performance (sci. comp & HPC) are the ones driving the (non-)adoption of PyPy. One, the existing solution (CPython + Cython) is pretty mature. Two, they require the holy trinity: Numpy, Scipy and Matplotlib. None of these are supported on PyPy, and only Numpy is even close.

Re:PyPy2.7 and PyPy3.5 v5.7 released by fijal on Tuesday March 21, 2017 @01:07PM (#13924118)
numpy works, scipy & matplotlib are very close. Stop spreading false information please.

That said, the cpython+cython solution is not going away any time soon.

Re:PyPy2.7 and PyPy3.5 v5.7 released by hyperbovine on Tuesday March 21, 2017 @03:27PM (#13925366)
> numpy works, scipy & matplotlib are very close.

Care to elaborate? I agree that numpypy is pretty much complete, though I have not found it to be faster in my limited testing. I was under the impression that a great deal of work would be required to get the other two running, even using CPyExt (which would not be faster, which was the whole point to begin with). I'd love to be proved wrong though.

Re:PyPy2.7 and PyPy3.5 v5.7 released by mattip on Tuesday March 21, 2017 @04:12PM (#13925773)
Real world use cases are large and messy. Now that the cython/pandas/numpy stack works out of the box on PyPy, we can begin to see how often the mix of python code (jittable, fast) and c code (fast) needs to cross the c-to-python barrier (slow). There may be some cases, like NLP where speeding up parsing and io can make a difference
Re:PyPy2.7 and PyPy3.5 v5.7 released by fijal on Wednesday March 22, 2017 @04:06AM (#13929206)
numpy, pandas (mostly) and (scipy) mostly work through cpyext. We're considering how to make them fast now. It would never be faster on large_array + large_array, but it should be faster (by a lot) with element access etc., bit like numba
Re:PyPy2.7 and PyPy3.5 v5.7 released by afarrell on Tuesday March 21, 2017 @08:00AM (#13921400)
What is the experience for someone trying it out for the first time and trying to install packages in it? What is the UX for someone building packages and verifying that they work?
Re:PyPy2.7 and PyPy3.5 v5.7 released by mikeash on Tuesday March 21, 2017 @10:15AM (#13922348)
I have a performance-critical Python program that I started running in pypy for an extra speedup. It was trivial: I downloaded the package, untarred it, and then ran my program with ~/Downloads/pypy-whateverversion/bin/pypy ./myprogram.py and it worked. Fancier stuff may require more effort, but the basics are really simple.
Re:PyPy2.7 and PyPy3.5 v5.7 released by rafinha on Tuesday March 21, 2017 @07:39AM (#13921314)
PIL
Re:PyPy2.7 and PyPy3.5 v5.7 released by LeifCarrotson on Tuesday March 21, 2017 @08:51AM (#13921688)
Expound, please?

Do you mean to refer to the Global Interpreter Lock as the factor which prevents it from becoming the official path for the language? CPython, the reference implementation, has the GIL as well.

Or the PIL library (though everyone seems to use Pillow now?)

Re:PyPy2.7 and PyPy3.5 v5.7 released by rafinha on Tuesday March 21, 2017 @10:53AM (#13922676)
oops, I meant GIL. I work with scientific computing. In my experience, the only justification I have of using other language (mostly C++) over python is to get over the interpreter lock, which over complicates your life if you want parallelisation. Multiprocessing is just awful, specially if you work with big chunks of data (I do mostly medical imaging). PyPy, even if it's faster, still has the same GIL constraints as CPython, so not really a big advantage IMO.
Re:PyPy2.7 and PyPy3.5 v5.7 released by fijal on Tuesday March 21, 2017 @01:08PM (#13924122)
We're working on it :-) In fact there are two projects - one is to just remove the GIL, one is to add STM-capabilities.
Re:PyPy2.7 and PyPy3.5 v5.7 released by leblancfg on Tuesday March 21, 2017 @07:42AM (#13921325)
For my (and I imagine many among us) use case, this is the takeaway:

PyPy2 can now import and run many C-extension packages, among the most notable are Numpy, Cython, and Pandas. Performance may be slower than CPython, especially for frequently-called short C functions. Please let us know if your use case is slow, we have ideas how to make things faster but need real-world examples (not micro-benchmarks) of problematic code.

Re:PyPy2.7 and PyPy3.5 v5.7 released by poizan42 on Tuesday March 21, 2017 @08:21AM (#13921502)
It would probably be better to port packages like Numpy to pure PyPy[0], but of course the CPython extensions compatibility layer is by itself important to increase the adoption of PyPy.

The problem besides the jit not being able to inline external functions is that the CPython structures, and especially the refcounting used in CPython is problematic [1]

[0]: Actually specifically numpy is being rewritten in PyPy as a RPython Mixed Module: http://doc.pypy.org/en/latest/extending.html#rpython-mixed-m... [1]: http://doc.pypy.org/en/latest/discussion/rawrefcount.html

Re:PyPy2.7 and PyPy3.5 v5.7 released by yedpodtrzitko on Tuesday March 21, 2017 @12:03PM (#13923446)
There's an attempt to create such NumPy fork already: https://bitbucket.org/pypy/numpy
Re:PyPy2.7 and PyPy3.5 v5.7 released by wfunction on Tuesday March 21, 2017 @07:52AM (#13921369)
If it's useful at all, PyPy 5.7.0 was noticeably slower than CPython 2.7.11 x64 when I was using SymPy. Haven't tried out the new release though.
Re:PyPy2.7 and PyPy3.5 v5.7 released by fijal on Tuesday March 21, 2017 @08:25AM (#13921522)
Hi

I've tried optimizing sympy and it's hard - there is a chance that it uses cython (which will slow things down on pypy) but also the gains were small (up to 2x) and only noticable if you run it for a while (at least few seconds).

SymPy is a bit of an example that while python as a language is not necesarilly slow, since you can make a plan for most parts, it breeds culture of unnecessary meta-programming, tons of copies and dict lookups that makes it REALLY HARD to optimize. ORMs go in the same category

Re:PyPy2.7 and PyPy3.5 v5.7 released by sanxiyn on Tuesday March 21, 2017 @08:22AM (#13921504)
Did you try x64 version of PyPy? x86 version is necessarily much slower (both CPython and PyPy) if your program is long-heavy. SymPy is actually part of benchmark used by PyPy and PyPy should be faster if you compare x64 to x64.
Re:PyPy2.7 and PyPy3.5 v5.7 released by wfunction on Tuesday March 21, 2017 @02:47PM (#13924994)
Nope, I'm on Windows.
Re:PyPy2.7 and PyPy3.5 v5.7 released by denfromufa on Tuesday March 21, 2017 @08:38AM (#13921603)
subset of sympy can run with symengine for speedup
Re:PyPy2.7 and PyPy3.5 v5.7 released by makmanalp on Tuesday March 21, 2017 @07:49AM (#13921360)
This is amazing! I had no idea we were so close to 3.x and cpyext. Congratulations to the pypy team for their brilliant work.

Perhaps it's almost time for some numpy / pandas stuff in speed.pypy.org :)

Re:PyPy2.7 and PyPy3.5 v5.7 released by bobbyi_settv on Tuesday March 21, 2017 @11:17AM (#13922923)
The Mac binary from the download page works perfectly for me. I tried a piece of numerical code I have been working on which makes some use of numpy but also uses element-by-element loops. The script takes 2 minutes with CPython, 3 minutes using pypy with NumPy via cpyext, and less than 2 SECONDS with pypy using NumPyPy. Wow!

However, when running the linux binaries on CentOS, they both (32- and 64-bit) fail for me with shared library errors:

./pypy2-v5.7.0-linux32/bin/pypy: error while loading shared libraries: libexpat.so.1: cannot open shared object file: No such file or directory

The 64-bit version gives the same library error but regarding libssl.so.1.0.0. I verified that I do have both expat and openssl installed (using yum), although I don't think that should be needed anyway. Anyone know what is happening here?

Re:PyPy2.7 and PyPy3.5 v5.7 released by collinmanderson on Tuesday March 21, 2017 @06:56PM (#13927043)
From the download page:

"[1]: stating it again: the Linux binaries are provided for the distributions listed here. If your distribution is not exactly [Ubuntu 12.04 and 14.04], it won't work, you will probably see: pypy: error while loading shared libraries: …. Unless you want to hack a lot, try out the portable Linux binaries." https://github.com/squeaky-pl/portable-pypy#portable-pypy-di...

Re:PyPy2.7 and PyPy3.5 v5.7 released by pettou on Tuesday March 21, 2017 @01:01PM (#13924056)
Congratulations and thank you!

One question to PyPy developers: CPython 3.5 [1] introduced math.gcd which uses Lehmer's algorithm [2], are there any plans to include it in PyPy, too?

[1] https://bugs.python.org/issue22486 [2] https://en.wikipedia.org/wiki/Lehmer%27s_GCD_algorithm

Re:PyPy2.7 and PyPy3.5 v5.7 released by fijal on Tuesday March 21, 2017 @01:07PM (#13924110)
Hi

Yes, absolutely. This is why it's marked as "beta", so not absolutely all the features are implemented.

Re:PyPy2.7 and PyPy3.5 v5.7 released by jsmeaton on Tuesday March 21, 2017 @08:26AM (#13921532)
What isn't supported by pypy? Thinking in terms of a modern Django app, what should I be looking out for? Is libxml supported? Database drivers like pyscopg?
Re:PyPy2.7 and PyPy3.5 v5.7 released by wyldfire on Tuesday March 21, 2017 @09:07AM (#13921806)
Very little, IMO. I'm rarely disappointed at whatever I throw at it. I have seen Django apps work correctly before and I would expect libxml and psycopg to work correctly too.
Re:PyPy2.7 and PyPy3.5 v5.7 released by danielsamuels on Tuesday March 21, 2017 @10:15AM (#13922350)
psycopg2cffi is available to provide Postgres support.
Re:PyPy2.7 and PyPy3.5 v5.7 released by jsmeaton on Tuesday March 21, 2017 @07:29PM (#13927250)
Thanks for that. I just ran a quick django shell test, and fetching 50,000 objects took 6.62 seconds in cpython, and 3.62 seconds in pypy. I haven't done a thorough test of the full app yet, but the basics appear to be working just fine.

Definitely something I'll be looking into more.

Re:PyPy2.7 and PyPy3.5 v5.7 released by kzrdude on Tuesday March 21, 2017 @01:28PM (#13924325)
My respect for pypy grows with them nearing CPython's current reality (3.x features). This is great news!
Re:PyPy2.7 and PyPy3.5 v5.7 released by mwcampbell on Tuesday March 21, 2017 @11:16AM (#13922909)
What is the current recommended MySQL driver to use with PyPy? And please don't tell me to use Postgres.
Re:PyPy2.7 and PyPy3.5 v5.7 released by ergo14 on Tuesday March 21, 2017 @11:16AM (#13922912)
Still, use postgres ;) There are a few pure-python drivers you can use if you need that (oursql?).
Re:PyPy2.7 and PyPy3.5 v5.7 released by wfunction on Tuesday March 21, 2017 @08:06AM (#13921434)
Will matplotlib ever be supported?
Re:PyPy2.7 and PyPy3.5 v5.7 released by fijal on Tuesday March 21, 2017 @08:23AM (#13921516)
yes, it "mostly works". There is an issue about graphic backends (you can either save it or use jupyter), but otherwise yes.
Re:PyPy2.7 and PyPy3.5 v5.7 released by wfunction on Tuesday March 21, 2017 @03:46PM (#13925524)
Whoops, I meant for Windows. I can't get it to install on Windows...
Re:PyPy2.7 and PyPy3.5 v5.7 released by ipunchghosts on Tuesday March 21, 2017 @09:44AM (#13922097)
is pyqt supported?
Re:PyPy2.7 and PyPy3.5 v5.7 released by fijal on Tuesday March 21, 2017 @12:33PM (#13923796)
no
Re:PyPy2.7 and PyPy3.5 v5.7 released by PunchTornado on Tuesday March 21, 2017 @09:02AM (#13921768)
pypy2.7? python 2 should just die.

Because of these people doing work for the ecosystem in 10 years we will still see this monstruosity: "for python2 do this and for python3 do this"

Re:PyPy2.7 and PyPy3.5 v5.7 released by fijal on Tuesday March 21, 2017 @12:34PM (#13923806)
well, I'm very sorry you wished we stopped supporting our users, but this is the reality which we'll be stuck in for a while. python 3 adoption has been speeding up, but still the vast majority of old big projects are based in python 2 and those are potential uses for pypy.
Re:PyPy2.7 and PyPy3.5 v5.7 released by sontek on Wednesday March 22, 2017 @01:37AM (#13928690)
Although I disagree with the sentiment that PyPy2/Python2 should die, I also disagree with the PyPy team treating PY3 as a second class citizen.

For this release PyPy2 shipped with binaries for 11 platforms, PyPy3 shipped with 1.

Re:PyPy2.7 and PyPy3.5 v5.7 released by mattip on Wednesday March 22, 2017 @01:48AM (#13928731)
Mozilla helped to solve this problem and one is orogress. Please encourage your company to donate to ooen source in general (maybe it already does?) and to PyPy in particular
Re:PyPy2.7 and PyPy3.5 v5.7 released by sontek on Wednesday March 22, 2017 @01:56AM (#13928762)
Thanks for the sentiment. We are major sponsors of PyCon and have donated to PyPy.

What I'm trying to suggest is that the amount they've received to support Python 3 ($250,000 from Mozilla, $66,677 in public donations) is signal that there is major demand for this.

Could they use the infra that is building binaries for PyPy2 for PyPy3?

I understand where they are coming from "Existing project are using Python2" but I'd like to see them focus more on the future.

I love what PyPy is doing and I want them to succeed but I can't imagine their success coming out of Python2. I see them more competing with more modern performant languages that are currently stealing from Python (i.e Go/Rust).

If they focused on the future they could directly compete in that market of projects that are rewriting in other languages for performance.