$ time python literal_translation.py phrase-table en test4
translation table contains 76885 elements
...................................................................... 070
.......................................................
125 files translated
real 1m14.345s
user 1m9.836s
sys 0m4.157s
$ time python3 literal_translation.py phrase-table en test5
translation table contains 76885 elements
...................................................................... 070
.......................................................
125 files translated
real 2m17.680s
user 2m2.605s
sys 0m4.987s
$ time pypy literal_translation.py phrase-table en test6
translation table contains 76885 elements
...................................................................... 070
.......................................................
125 files translated
real 1m13.347s
user 1m0.033s
sys 0m4.734s
https://gist.github.com/andreasf/af6bdc00cf5a928712b5Memory-mapping a serialisation of the dict would help, but I'm not sure how you would do that outside of an extension lib.
But did you try using it as a service (ie. avoid the startup time) and timing the actual translation?
At work, we switched to pypy for the speed. For a spell-checker I made in python, pypy made the dictionary compiling 3x faster and the spelling 2x faster, so it can really matter. On the other hand, pypy's memory usage is often a bit worse, and startup time can also be a bit slower, at least if your program has an otherwise "zero" startup time.
If it's worth the effort, use Cython, design your data structures carefully, and you'll get C level performance.
Rather, when you do the test, only test the "mapping" part, and only test the "IO" part. That way you can clearly see the difference between the two.
However, we're more than happy to consider it a pypy bug if we're not faster than CPython. Please provide clearer instructions how to reproduce it and ideally also other benchmarks that did not work for you.
The script and data are available here: http://zentrale1.com/~an/literal-translation.tar.xz
The texts were scraped from the European Parliament web site. The phrase table was built with Moses from the Europarl parallel corpus [0] minus the texts.
The script itself was supposed the be baseline input for another experiment, basically it's the worst way of translating something i could think of.
It seems like it's often much faster than PyPy (up to 4x) and almost never slower.