News for hackers. Stuff that matters.
Re:Four MLs (and a Python) by unhammer on Thursday April 30, 2015 @03:00AM (#9463675)

    $ time wc big-data.csv                                                                     
     1252045  1252045 17805309 big-data.csv
    
    real    0m0.417s
    user    0m0.411s
    sys     0m0.004s

    $ time ./sum-ocaml big-data.csv                                                            
    6143290.,13087196.,14473656.,50128757.,3765822.,1290420.
    
    real    0m3.680s
    user    0m3.666s
    sys     0m0.008s


    $ time python3 sum.py big-data.csv          
    6143290.0,13087196.0,14473656.0,50128757.0,3765822.0,1290420.0
    
    real    0m7.186s
    user    0m7.131s
    sys     0m0.036s

    $ time python2 sum.py big-data.csv         
    6143290.0,13087196.0,14473656.0,50128757.0,3765822.0,1290420.0
    
    real    0m5.146s
    user    0m5.111s
    sys     0m0.012s

    $ time pypy2 sum.py big-data.csv            
    6143290.0,13087196.0,14473656.0,50128757.0,3765822.0,1290420.0
    
    real    0m2.754s
    user    0m2.697s
    sys     0m0.052s
I really like ocaml, but have to admit, pypy is pretty amazing.
Re:Four MLs (and a Python) by andor on Thursday April 30, 2015 @05:09AM (#9464000)
It completely depends on the application. I have yet to see a use case that benefits from PyPy. Here's an example I tried with CPython2, 3 and PyPy yesterday: It loads the most probable translations for all unigrams from a Moses phrase translation table (11GB) into a dict, and using that dict translates 73 MB of text word by word. Most of the time is spent filling the dict/reading the large file.

  $ time python literal_translation.py phrase-table en test4
  translation table contains 76885 elements
  ...................................................................... 070
  .......................................................
  125 files translated
  
  real	1m14.345s
  user	1m9.836s
  sys	0m4.157s
  
  
  $ time python3 literal_translation.py phrase-table en test5
  translation table contains 76885 elements
  ...................................................................... 070
  .......................................................
  125 files translated
  
  real	2m17.680s
  user	2m2.605s
  sys	0m4.987s
  
  
  $ time pypy literal_translation.py phrase-table en test6
  translation table contains 76885 elements
  ...................................................................... 070
  .......................................................
  125 files translated
  
  real	1m13.347s
  user	1m0.033s
  sys	0m4.734s

https://gist.github.com/andreasf/af6bdc00cf5a928712b5
Re:Four MLs (and a Python) by 314 on Thursday April 30, 2015 @06:38AM (#9464223)
If your problem is memory-bound then no compiler will help you.

Memory-mapping a serialisation of the dict would help, but I'm not sure how you would do that outside of an extension lib.

Re:Four MLs (and a Python) by unhammer on Thursday April 30, 2015 @07:05AM (#9464295)
1m for 11GB sounds IO-bound, but yeah that's still not a use-case where switching to pypy will make a difference.

But did you try using it as a service (ie. avoid the startup time) and timing the actual translation?

At work, we switched to pypy for the speed. For a spell-checker I made in python, pypy made the dictionary compiling 3x faster and the spelling 2x faster, so it can really matter. On the other hand, pypy's memory usage is often a bit worse, and startup time can also be a bit slower, at least if your program has an otherwise "zero" startup time.

Re:Four MLs (and a Python) by andor on Thursday April 30, 2015 @07:36AM (#9464395)
No, I didn't try to optimize this any further. I just checked whether any of the new interpreters is really faster. It's a simple one-off experiment, and compared to real statistical machine translation with Moses on the same data the translation is pretty much instant. Moses needs something on the order of 1600 CPU-hours (50 nodes * 8 threads * 4 hours).
Re:Four MLs (and a Python) by syllogism on Thursday April 30, 2015 @08:32AM (#9464622)
NLP stuff tends to be more data-bound, which PyPy can't really help with.

If it's worth the effort, use Cython, design your data structures carefully, and you'll get C level performance.

Re:Four MLs (and a Python) by andor on Friday May 01, 2015 @08:09AM (#9470772)
Hey, thanks for spaCy! It's a great library.
Re:Four MLs (and a Python) by zo1 on Thursday April 30, 2015 @02:09PM (#9466664)
This isn't an isolated test, and tells us almost nothing. Because you've clumped together IO as well as the bulk of the actual processing work.

Rather, when you do the test, only test the "mapping" part, and only test the "IO" part. That way you can clearly see the difference between the two.

Re:Four MLs (and a Python) by GFK_of_xmaspast on Thursday April 30, 2015 @08:22AM (#9464575)
I notice python3 is twice as slow.
Re:Four MLs (and a Python) by andor on Thursday April 30, 2015 @07:45AM (#9464426)
To whoever downvoted this -- how about posting a more comprehensive set of benchmarks instead? My single benchmark is not in any way less meaningful than the parents. But as I said, I often try PyPy on longer-running Python code, and so far saw no significant gains.
Re:Four MLs (and a Python) by fijal on Thursday April 30, 2015 @09:52AM (#9465020)
Your benchmark is terrible - it's impossible to run it because you don't provide a dataset that you used (or I'm missing some description). It also stresses one aspect of the language (processing strings, in a very specific way) and yet you claim this is representative.

However, we're more than happy to consider it a pypy bug if we're not faster than CPython. Please provide clearer instructions how to reproduce it and ideally also other benchmarks that did not work for you.

Re:Four MLs (and a Python) by andor on Friday May 01, 2015 @01:36PM (#9473117)
Great, next time I'll hit your bug tracker directly ;-)

The script and data are available here: http://zentrale1.com/~an/literal-translation.tar.xz

The texts were scraped from the European Parliament web site. The phrase table was built with Moses from the Europarl parallel corpus [0] minus the texts.

The script itself was supposed the be baseline input for another experiment, basically it's the worst way of translating something i could think of.

[0] http://statmt.org/europarl/

Re:Four MLs (and a Python) by fijal on Monday May 04, 2015 @06:21AM (#9485273)
can you please provide me the data in somehow reproducible form (I have no clue how to use all the tools you mentioned for example), ideally as a .tar.bz2 file somewhere I can download and run without any prior knowledge. I'm happy to improve pypy for that particular use case.
Re:Four MLs (and a Python) by brendano on Thursday April 30, 2015 @01:22PM (#9466356)
"Terrible" is a little harsh. I found the benchmark to be interesting because it's similar to many NLP-style analysis tasks I have myself. Of course reproducible ones would be great too.
Re:Four MLs (and a Python) by pron on Thursday April 30, 2015 @04:17AM (#9463868)
And ZipPy -- a Python implementation for the JVM based on Truffle/Graal -- supposedly performs even better than pypy.
Re:Four MLs (and a Python) by fijal on Thursday April 30, 2015 @09:55AM (#9465034)
[citation needed]. In the minute or so of idle browsing I did not find that claim on their website nor a claim how much python they actually implement.
Re:Four MLs (and a Python) by pron on Thursday April 30, 2015 @10:57AM (#9465441)
https://docs.google.com/spreadsheets/d/1fFMWcRIuPKt7wSAM5Ox9...

It seems like it's often much faster than PyPy (up to 4x) and almost never slower.

Re:Four MLs (and a Python) by cwyers on Thursday April 30, 2015 @01:52PM (#9466558)
I feel like if a program runs for less than 10 seconds, I don't much care if it's 2 seconds or 7 seconds. And a benchmark for a program like that tells me very little about how the various runtimes affect larger programs.