News for hackers. Stuff that matters.
Re:PyPy Project looking for sponsorship to add support for A... by mattip on Thursday December 31, 2020 @09:56AM (#25593193)
For those asking "who uses PyPy?": the truth is that very few people using it get back to us and say how it helps them or what they are doing with it. For instance crossbar.io uses it [0]. Some shops use it as a second step until they refactor code into a compiled language, but often that second step never materializes.

But the value of a second implementation of a language goes beyond the immediate "who uses this in practice". It can be a fertile bed for innovation and for new ideas, and provides a contrast to the nay sayers. For instance, the recent pitch to vastly improve CPython's speed has some roots in ideas that were tested out in PyPy. CFFI [1], revdb [2] and vmprof [3] all started as PyPy projects. Some of these turned out to be very popular, some less so. The next project in this line is HPy [4] (still alpha-quality), which is trying to re-think the C-API for Python to make it even easier to interface with.

RPython [5], the language behind PyPy, is also an accessible playground for dynamic language research.

[0] https://crossbar.io/about/FAQ/#python-runtime

[1] https://cffi.readthedocs.io/en/latest/

[2] https://morepypy.blogspot.com/2016/07/reverse-debugging-for-...

[3] https://vmprof.readthedocs.io/en/latest/

[4] https://hpy.readthedocs.io/en/latest/

[5] https://rpython.readthedocs.io/en/latest/examples.html

Re:PyPy Project looking for sponsorship to add support for A... by ghj on Thursday December 31, 2020 @01:39PM (#25595590)
If you want more examples of real world use cases, PyPy is pretty stress-tested by the competitive programming community already.

https://codeforces.com/contests has around 20-30k participants per contest, with contests happening roughly twice a week. I would say around 10% of them use python, with the vast majority choosing pypy over cpython.

I would guesstimate at least 100k lines of pypy is written per week just from these contests. This covers virtually every textbook algorithm you can think of and were automatically graded for correctness/speed/memory. Note that there's no special time multiplier for choosing a slower language, so if you're not within 2x the speed of the equivalent C++, your solution won't pass! (hence the popularity of pypy over cpython)

The sheer volume of advanced algorithms executed in pypy gives me huge amount of confidence in it. There was only one instance where I remember a contestant running into a bug with the jit, but it was fixed within a few days after being reported: https://codeforces.com/blog/entry/82329?#comment-693711 https://foss.heptapod.net/pypy/pypy/-/issues/3297.

Re:PyPy Project looking for sponsorship to add support for A... by dragonwriter on Thursday December 31, 2020 @03:18PM (#25596683)
> If you want more examples of real world use cases, PyPy is pretty stress-tested by the competitive programming community already.

I think this is the first time I've seen someone suggest that competitive programming has much bearing on “real-world use cases”.

Re:PyPy Project looking for sponsorship to add support for A... by saagarjha on Thursday December 31, 2020 @06:32PM (#25598516)
I guess it tells you that loops and basic containers work?
Re:PyPy Project looking for sponsorship to add support for A... by andrewchambers on Thursday December 31, 2020 @06:34PM (#25598543)
So essentially all code right?
Re:PyPy Project looking for sponsorship to add support for A... by saagarjha on Thursday December 31, 2020 @08:03PM (#25599480)
Right, but most code does things other than that as well.
Re:PyPy Project looking for sponsorship to add support for A... by gabagool on Thursday December 31, 2020 @02:33PM (#25596225)
Why do competitive Python programmers use PyPy instead of CPython?
Re:PyPy Project looking for sponsorship to add support for A... by ghj on Thursday December 31, 2020 @03:13PM (#25596638)
PyPy is easily 10x faster than CPython at numeric stuff, which is 99% of these contest problems.

For example, using CPython, if you try to make an array of a million ints, you won't get `int[1000000]` in your memory layout. Each int would actually be an object, which is huge and inefficient to reference (they are something like 24+ bytes each).

PyPy on the other hand, works as expected.

I think the more important point is that PyPy when written like C code, can actually get within 2x of the performance of C code. If it's any slower, python won't be a viable language in competitive programming at all.

(CPython is sometimes still used on other platforms like atcoder.jp, but only because they allow third party libraries like numba and numpy which can fill the same role pypy does)

Re:PyPy Project looking for sponsorship to add support for A... by simple_phrases on Thursday December 31, 2020 @05:55PM (#25598165)
For that particular use case, how does PyPy perform in comparison to CPython's array module[1]?

[1] https://docs.python.org/3/library/array.html

Re:PyPy Project looking for sponsorship to add support for A... by spicybright on Thursday December 31, 2020 @08:31PM (#25599791)
Why doesn't the community collectively switch over to PyPy? It seems like it's better in all reguards.
Re:PyPy Project looking for sponsorship to add support for A... by semi-extrinsic on Thursday December 31, 2020 @08:59PM (#25600012)
Library support?
Re:PyPy Project looking for sponsorship to add support for A... by sitkack on Thursday December 31, 2020 @09:17PM (#25600150)
The cabal that runs CPython ignores both standardization, the standard library and interop with other implementations.

PyPy shouldn't be the new default, but neither should be CPython.

Re:PyPy Project looking for sponsorship to add support for A... by spicybright on Friday January 01, 2021 @02:06PM (#25605748)
Don't you have to choose one or the other tho lol
Re:PyPy Project looking for sponsorship to add support for A... by neolog on Thursday December 31, 2020 @02:55PM (#25596420)
PyPy is much faster than CPython.
Re:PyPy Project looking for sponsorship to add support for A... by Guthur on Thursday December 31, 2020 @05:37PM (#25597975)
For some cases, and the picture often changes.

We were using pypy because it was better for our use case at one point but then later on we retested cpython and found the picture had changed.

We believe this was due to significant improvements in the regex engine for cpython over the period, but could also be due to our code base changing.

The point being it is not a given that pypy is faster.

Re:PyPy Project looking for sponsorship to add support for A... by Recursing on Thursday December 31, 2020 @10:41AM (#25593628)
AFAIK pypy improvements also led to cpython compact dictionaries (which led to dictionaries being ordered) and many many cpython performance improvements (the most recent one being LOAD_ATTR caching, with huge performance gains).

I wonder if sponsoring pypy instead of the PSF might be a way for people that want "a focus on performance improvements, instead of fancy features" to vote with their wallets

Re:PyPy Project looking for sponsorship to add support for A... by ngfellow on Thursday December 31, 2020 @11:45AM (#25594248)
It is worth noting that none of the PSF donations arrive at core development anyway. They support the bureaucracy and the infrastructure.

python.tar.gz would be exactly the same if the donations were zero.

All the more reason to switch to donating PyPy.

Re:PyPy Project looking for sponsorship to add support for A... by tzl on Thursday December 31, 2020 @12:19PM (#25594645)
Instead of downvoting, google the financial statements and see for yourself. You are supporting bureaucracy/infra/conferences.

All of which would not exist without python.tar.gz.

Re:PyPy Project looking for sponsorship to add support for A... by dmw_ng on Thursday December 31, 2020 @12:27PM (#25594750)
Going by https://www.python.org/psf/annual-report/2020/ , 75% are running the annual conference, 12.6% grants open to _anyone_, and 3.8% maintaining a central registry primarily staffed by volunteers, and I imagine paying for quite a sizeable set of origin servers at this stage.

As for the PyCon expenditure, it's certainly the kind of thing I'd never attend, but there can be no doubt the existence of the conference has strengthened and grown Python's user base, and it is open to everyone should they wish to attend. Meanwhile, $97k annual to operate a package registry with literally millions of downloads per month that tens of thousands of companies depend on sounds like an amazing bargain to me.

Re:PyPy Project looking for sponsorship to add support for A... by incongruity on Thursday December 31, 2020 @01:18PM (#25595352)
Speaking as someone who had a core role in organizing PyCon US a decade+ ago and was active with the PSF for some time, I’ve never seen an organization more dedicated to good stewardship and inclusiveness at scale. I’m sure there are other great examples but the PSF was always commendable.

PyCon was always meant to be affordable for as many people as it could be - via low prices and financial aid for many. Those conferences helped create many of the relationships that drove key parts of the Python ecosystem. In addition, there are/were typically dedicated time for code sprints following the main conference. Important work happened there, often for key parts of the Python codebase or key libraries. While the budget numbers don’t say explicitly “development support” people shouldn’t presume PyCon does nothing for the language itself. Open Source conferences are not like commercial conferences. It’s community driven rather than marketing driven. That difference matters.

It was a smaller world when I was involved but it was and still is something special.

Re:PyPy Project looking for sponsorship to add support for A... by marmaduke on Thursday December 31, 2020 @04:19PM (#25597259)
> literally millions of downloads per month

I was surprised it would be so low given all the Dockerfiles building in CI systems everywhere and apparently from

https://pypistats.org/top

It looks like tens of millions per day a billions per month

edit which is ~115 downloads per second, which is nothing a beefy VM couldn’t handle.

Re:PyPy Project looking for sponsorship to add support for A... by dmw_ng on Thursday December 31, 2020 @07:22PM (#25599012)
BigQuery:

    #standardSQL
    SELECT EXTRACT(HOUR FROM timestamp) hour, COUNT(*) AS num_downloads
    FROM `the-psf.pypi.file_downloads`
    WHERE DATE(timestamp)
        BETWEEN DATE_SUB(CURRENT_DATE(), INTERVAL 30 DAY)
        AND CURRENT_DATE()
    GROUP BY hour
    ORDER BY num_downloads
Gives us 6.956 billion downloads in the past 30 days

With hour 15 clocking in at around 2,964 requests/sec -- mean. This is still not representative of peak instantaneous traffic, which I'd hazard a guess is somewhere closer to 10x that number.

Now factor in the 'usual' case where the CDN absorbs 70% of this, and you're still left with a service that is non-trivial to run and support 24/7. PyPi manages to only have a few small downtimes per year.

Re:PyPy Project looking for sponsorship to add support for A... by marmaduke on Friday January 01, 2021 @04:59PM (#25607640)
Hadn’t thought to look up more precise numbers, and the comment on CDN is good one. It’s an example of where smart choices in the client (pip) could keep the backend (PyPI) scalable.
Re:PyPy Project looking for sponsorship to add support for A... by wrycoder on Thursday December 31, 2020 @06:48PM (#25598711)
PyCon makes a sizeable profit, which supports the staff and grantmaking activities of the PSF.
Re:PyPy Project looking for sponsorship to add support for A... by oefrha on Thursday December 31, 2020 @12:39PM (#25594864)
PyCon totally does affect python.tar.gz.

https://github.com/python/cpython/pulls?q=+label%3Asprint (and that link doesn’t capture the PEPs hashed out among other things.)

Also, why the hell make a new account for each comment?

Re:PyPy Project looking for sponsorship to add support for A... by nerdponx on Thursday December 31, 2020 @12:33PM (#25594801)
Here's the 2020 financial statement: https://www.python.org/psf/annual-report/2020/

Total expenses for 2019 were a bit under $3.7m. "Program service expenses" were $2.6m, about 70% of total expenses. Of this $2.2m, 75.3% was spent on PyCon. So overall, around 53% of PSF spending in 2020 went to PyCon ($1.9m). In 2018, it was around 57%.

The next-biggest expense was staffing ($694k), at around 19% in 2019.

Re:PyPy Project looking for sponsorship to add support for A... by bobajeff on Thursday December 31, 2020 @10:24AM (#25593454)
To add I believe PyPy also inspired asm.js the precursor to Web Assembly.
Re:PyPy Project looking for sponsorship to add support for A... by cgearhart on Thursday December 31, 2020 @08:33PM (#25599805)
I have turned to pypy quite extensively for pure-Python text processing tasks, where I often get a 10-100x speed up just by changing the command line invocation. For example, I wrote a proof of concept Rabin-Karp hashing approximate string matching algorithm to perform plagiarism analysis while I was working at Udacity in around 2017. The system never went into production, but pypy was super helpful in crunching all historical user submissions for analysis.

I’ve also had great success using pypy to accelerate preprocessing steps (when they don’t rely on incompatible c libraries) for machine learning pipelines. It’s the most painless performance enhancement trick in my toolbox before I reach for concurrency (in which case I reach for joblib or Dask).

The one oddity I’ve noticed is that using tuples (including things like named tuples) often speeds up CPython by a lot, but even plain tuples can slow down pypy on the same code—in some cases pypy winds up slower than CPython.

In any case, I’m low key in love with pypy, even though I can’t use it for _everything_.

Re:PyPy Project looking for sponsorship to add support for A... by sitkack on Thursday December 31, 2020 @09:20PM (#25600171)
> The one oddity I’ve noticed is that using tuples (including things like named tuples) often speeds up CPython by a lot, but even plain tuples can slow down pypy on the same code—in some cases pypy winds up slower than CPython.

I'd love for this to be addressed. namedtuple should be lowered into a fixed sized struct, great opportunity for PyPy to show wins in both memory, memory bandwidth and compute. Add in type stability and PyPy should approach native speeds just by adding in namedtuple.

Re:PyPy Project looking for sponsorship to add support for A... by mattip on Friday January 01, 2021 @12:35AM (#25601151)
I created an issue [0] to address this. Could you add a microbenchmark that demonstrates the problem?

[0] https://foss.heptapod.net/pypy/pypy/-/issues/3373

Re:PyPy Project looking for sponsorship to add support for A... by olliemath on Friday January 01, 2021 @08:42AM (#25603184)
We use pypy pretty extensively at our firm for analytics/olap work.

We've actually tried using some of the more traditional libs (Pandas et al) with CPython, but there's always a pure-python bottleneck (e.g. SQLAlchemy).

Performance is important to our clients and trying to keep everything performance critical in C extensions / NumPy would be kind of risky for us when adding new functionality, so pypy's guarantee of more speed pretty much across the board is awesome.

There are downsides of course - higher memory usage, longer boot times, some more obscure libraries being unsupported - but on the whole, it's a good choice for us

Re:PyPy Project looking for sponsorship to add support for A... by dheera on Thursday December 31, 2020 @02:25PM (#25596146)
Why isn't Ubuntu shipping it as the default python?
Re:PyPy Project looking for sponsorship to add support for A... by wongarsu on Thursday December 31, 2020 @02:44PM (#25596317)
Some popular libraries don't work on PyPy because of their C code.