News for hackers. Stuff that matters.
Re:Optimizing Python – A Case Study by lqdc13 on Friday May 29, 2015 @02:38AM (#9623092)
First thing you should do is optimize data structures IMO.

This is the advantage Python has over lower level languages - easy way to implement complicated things.

Kind of like Linus's quote: "Bad programmers worry about the code. Good programmers worry about data structures and their relationships."

Re:Optimizing Python – A Case Study by sitkack on Friday May 29, 2015 @06:30AM (#9623736)
I would say it is a close second thing. If I have a slow system, I will move to faster runtime before modifying any code. Going from CPython to PyPy, if possible will almost always gain you enough perf increase while you refactor the slow parts.
Re:Optimizing Python – A Case Study by fnord123 on Friday May 29, 2015 @07:25AM (#9623902)
It depends on what you're doing. I think your strategy can often be high risk since you are making a change that affects every part of your program at once -changing the runtime on a mature Python project is not a trivial choice! Changing the relevant code to tidy up data structures, or moving a few function calls to Cython or C, however, will only touch the obvious places and so it has less impact on the program.

Also, profiling pypy is less straightforward than profiling CPython code since the hotspot changes the runtime characteristics of the program. This means you need to run tests many times over to make sure the code warms up. This makes further optimizations slightly more difficult. It's not a problem for people with experience optimizing Python code, but for people who actually hope to learn something from OP's blog post, it might be a sticking point.

In my experience in using Python for stats, script type work (as opposed to writing servers and daemons), pypy just isn't that useful. All the Python code is doing is gluing numpy and Cython code together and pypy isn't likely to be able to warm up in time to beat it - and it won't beat it since it's spending most of its time in C.

Obviously, if pypy is an ideal choice for you, use it. But I don't think your experience should really be put forward as a general approach.

Re:Optimizing Python – A Case Study by sitkack on Friday May 29, 2015 @08:58AM (#9624238)
I have found PyPy wins if the wall time is over about 1.5 seconds, making it my goto Python environment for 1GB+ ETLs.

Everything you say is true, but under the systems I use/write I am able to test for correctness pretty quickly. The nice thing about switching from CPython to PyPy is that everything get faster. I have also found that using PyPy has removed lots of cases where I would want to drop down to native code.

Changing platforms can make one's designs simpler and more robust. When it comes to structured storage, I'll start with sqlite, then when it starts to get slow I'll switch to PostgreSQL. It takes almost no work to port from one to the other.

You really should give PyPy another shot. It supports more of numpy every day and the startup time is excellent. Maybe give jitpy a try if you are not likely to move off of CPython.