http://alexgaynor.net/2011/apr/03/my-experience-computer-lan...
Sounds entirely reasonable to me.
They seemed to have addressed most of the fairness and "What about language X on platform Y using framework Z" kinds of questions with "Send us a pull request and we'll include it in the next benchmark".
I'm tempted to tackle this even though I don't currently have the time.
[1] http://www.techempower.com/benchmarks/ [2] https://github.com/TechEmpower/FrameworkBenchmarks
I believe they have plans to add additional tests which will allow for caching and that kind of thing.
I don't think that removing caching across all frameworks equally is creating 'artificially hamstrung implementations' as all the frameworks would benefit from caching.
I'm not saying it would be easy (or economically viable), but it's certainly possible. Somebody should try it.
This isn't an automateable task, although it may be a crowdsource-able one. Luckily the benchmark game publishes pretty much all of the tooling they use, so you could probably get a pretty big headstart there on your improvements.
NOT TRUE.
Programs optimized for PyPy were accepted and shown alongside programs written for CPython. Joe LaFata contributed excellent PyPy pi-digits, spectral-norm, mandelbrot programs and they were all accepted.
>>banning multiple implementations<<
NOT TRUE.
Measurements for both JRuby and CRuby are shown.
Like everyone else, I'm sitting on my hands waiting for someone else to make and publish all the other program measurements.
The implementation pick ranges from the standards that everyone uses to the experimental that nobody but the language developer uses.
That's sad. Someone should fork it... would be great.
Another good benchmark that compares them all: http://www.techempower.com/benchmarks/#section=data-r8&hw=i7... (found it on HN some month ago)
I know maintaining a project like that is a lot of work. But adding superior implementations of already featured languages should be a piece of cake. Source code of Python, PHP and Lua is already in the repo, it's just a matter of installing the VM's on the computer and modify the scripts.
Beside, if the maintainer reads this, please keep up the good work, and maybe add LuaJIT, PyPy and HHVM. Thanks.
PyPy was shown (on my initiative) and I asked for (and published) programs that would work with PyPy. For example check the comments here -- http://morepypy.blogspot.com/2010/03/introducing-pypy-12-rel...
LuaJIT was also shown; HHVM was never shown.
http://web.archive.org/web/20110125034347/http://shootout.al...
That can be a pretty huge "it's just". In practice many of the existing scripts make use of C bindings or rely on odd behaviors that aren't replicated by alternate VMs. Since the alternate implementation, benchmark programs, and benchmark tools are all available, I'd recommend you put your theory to the test. It'll either prove eye opening or make for a nice contribution back to the game!
HHVM is compatible with PHP 5.3, LuaJIT with Lua 5.1 and PyPy with Python 2.7.3. All benchmark code is written the languages code and sometimes relies on an language extension library (which is written in C or C++). But three above JIT VMs come with their own compatible language extension libraries or support the extension API.
Look at the code, it should be easy: http://benchmarksgame.alioth.debian.org/u32q/benchmark.php?t... and https://github.com/kragen/shootout/tree/master/bench
Otherwise people might get the impression that C is the "speed of light" as far as program speed is concerned.