> The lesson is clear: do not write Python code as you would do in C. Use numpy array operations rather than iterate on arrays. For me it meant a mental shift.
That's true in C as well. If you're writing performance-critical code, you use things like BLAS and LAPACK (possibly even the optimized, often parallel versions supplied by your compiler vendor).
Thank you, you made Pypy move higher on my todo list.
You have a good point about gc and memory layout. That's why I like C or C++ when writing high performance software. You can control memory locality more easily than with other languages.
Sure, but what if my work depends on a library that uses C extensions that isn't supported by pypy? Any of the scikits, for example.
> The lesson is clear: do not write Python code as you would do in C. Use numpy array operations rather than iterate on arrays. For me it meant a mental shift.
I think the lesson is that "performance critical software requires attention to very many specific details." Using a garbage collected language that allocates objects on the heap by default means that you'll have to take special steps to change the default behavior of the system. You will see very similar behavior with C++, Java, etc.
> I didn't use Pypy so far because it only supports part of Numpy, and not Pandas, Scipy, and Sklearn. It may be that Pypy supports enough of Numpy to run my closest distance code though. If that's the case, then I'll blog about it soon.
I'll confess, I was eagerly waiting the pypy results. IMO using pypy means I get the best of both worlds: more code that can be naively python with python idioms and still yield sufficient performance. For critical algorithms where I can't hit sufficient performance, it's appropriate to use cffi or something to optimize.