http://alexgaynor.net/2011/apr/03/my-experience-computer-lan...
Sounds entirely reasonable to me.
They seemed to have addressed most of the fairness and "What about language X on platform Y using framework Z" kinds of questions with "Send us a pull request and we'll include it in the next benchmark".
I'm tempted to tackle this even though I don't currently have the time.
[1] http://www.techempower.com/benchmarks/ [2] https://github.com/TechEmpower/FrameworkBenchmarks
I believe they have plans to add additional tests which will allow for caching and that kind of thing.
I don't think that removing caching across all frameworks equally is creating 'artificially hamstrung implementations' as all the frameworks would benefit from caching.
I'm not saying it would be easy (or economically viable), but it's certainly possible. Somebody should try it.
This isn't an automateable task, although it may be a crowdsource-able one. Luckily the benchmark game publishes pretty much all of the tooling they use, so you could probably get a pretty big headstart there on your improvements.
NOT TRUE.
Programs optimized for PyPy were accepted and shown alongside programs written for CPython. Joe LaFata contributed excellent PyPy pi-digits, spectral-norm, mandelbrot programs and they were all accepted.
>>banning multiple implementations<<
NOT TRUE.
Measurements for both JRuby and CRuby are shown.
Like everyone else, I'm sitting on my hands waiting for someone else to make and publish all the other program measurements.
The implementation pick ranges from the standards that everyone uses to the experimental that nobody but the language developer uses.