How about...no?

PyPy is a nice way to make most Python programs 20-100% faster, often at the cost of using slightly more memory. Writing in C/C++/Cython etc, we typically see 100x run-time and memory improvements.

These extensions were written in C for a reason. If I had to write in pure Python, I'd find another language for the bulk of my work.

The reason is that PyPy didn't exist, and C extensions were the way to go back then. That is no longer true. Anything else is misinformation.

I don't use numpy (or really very many scientific libraries), and talking about generalities like you mentioned just isn't meaningful, but here's a really old benchmark which is (small) evidence that it's just not true that these things need to be written in C anymore: http://morepypy.blogspot.com/2012/01/numpypy-progress-report...

Why are you trying to push this point, when you plainly admit you don't know much about this? And so strongly, too! "Anything else is misinformation." I've seldom seen someone so confidently wrong.

PyPy can address Python's slow loop performance and slow numeric calculations, in particular examples. This can help a lot. But if the problem requires you to carefully manage your data structures so that you get good cache performance, PyPy cannot help much at all.

Consider this: there are very few non-toy examples of PyPy giving order of magnitude efficiency improvements over CPython, while other languages do typically run that much faster than Python implementations.

Finally, if you don't want to talk about generalities...Consider my NLP library, spaCy: http://honnibal.github.io/spaCy/#speed-comparison . It's the fastest library with this functionality in the world, among any language.

This library is an example where you need C data structures just to make the working set manageable. You would need an enormous amount of memory just to load the models. In pure Python, the library would not be of much use to anyone --- so I would have written it in Java or C++.

I did not admit I don't know much about this, although I will now, but that's not the important thing in these discussions. Very few people in the scientific community know what's out there, and many are misinformed in the exact same ways.

You seem to be suggesting that everything is fine, and that all the C code that exists right now in the scientific Python community is justified. This is incorrect.

Needing "C data structures" (I'm guessing you mean memory management but I'd love to have a look at your library tomorrow morning) is certainly reasonable -- and there are better ways to do so than the CPython API.

My point before, which I am pushing, yes, is that the scientific Python community is using old tools, and they'd benefit from using new ones that actually would directly benefit them, which PyPy will -- CFFI is a better choice than ctypes or the CPython API for way more agreeable reasons than whether Python 3 is a better choice than Python 2, and pure-python is a better solution than both for where it's applicable -- it's unimportant where that is to me, my point is "it's more than what we currently have in pure-python".

Numba is faster than pypy and compatible with cpython.

The scientific community will use whatever makes it easier to achieve its own goals. The idea that you need to use something because it is cooler, more open, fancier, etc., works only in the programming community, where software itself is the end goal.

At a minimum, that would require largely rewriting scipy, pandas, and any other piece of software that uses the CPython C extension interface rather than the FFI.

That's a pretty tall order...

SciPy and Pandas use Cython, so if that can ever be made runnable semi-directly in PyPy, those two are dealt with. Only NumPy uses the CPython API directly but this is being worked on as NumPyPy (see http://morepypy.blogspot.de/2015/02/linalg-support-in-pypynu...).

SciPy uses Cython, but it also uses other C interfaces. The most definitive statement I found in a couple minutes is here:

http://www.scipy.org/scipylib/faq.html#how-can-scipy-be-fast...

SciPy uses a variety of methods to generate “wrappers” around these algorithms so that they can be used in Python. Some wrappers were generated by hand coding them in C. The rest were generated using either SWIG or f2py. Some of the newer contributions to SciPy are either written entirely or wrapped with Cython.

> f2py

This is a Fortran-to-Python interface, and has some support for Python 3 so far as I can tell.

Actually, most of the time, the performance-critical computations happens in the highly optimized C modules already, so there's not /that/ much to gain. That being said, the PyPy team is working on a NumPy integration (NumPyPy).

In many cases, you can use something like python-bond to call PyPy from a regular CPython program: https://pypi.python.org/pypi/python-bond (or just use multiprocessing).

The C extensions shouldnt be cut out of Python. I get that PyPy is better but cutting out c sxtensions and rewriting them in pure python would hurt performance more.

There are several other options. The extension can be written as a pure C extension, with CFFI for the bindings, and code in Python to present the public API. Or write the extension in Cython, which has backends for both CPython and PyPy's cpyext (I have no experience with this option).

Personally, I would really like to see more Python extensions written using the ctypes foreign function library. This has two advantages:

* Supported on more than just CPython. I.e. you can use them on PyPy as well.

* The extensions are far easier to install, since one doesn't need both a full C compiler as well as all development headers for the library installed on the server where you install the extension.

ctypes overhead is horrendous though, at least in CPython (I don't know about PyPy). Fine if you're doing array operations on huge arrays, but not so if you have lots of small objects.

I wanted to use ctypes to wrap a C library for a recent project (for the ease of installation and development that you mention) but had to give up when it turned out to be more than 10x slower than a wrapper written in Cython, and barely faster than doing a pure Python implementation of the C library itself.

It's unclear that that's true (that performance would be worse), but regardless, there are many other, better ways to run C code from within Python (CFFI being the easiest and nicest currently)