News for hackers. Stuff that matters.
Re:Rust on Computer Language Benchmarks Game by masklinn on Thursday February 13, 2014 @02:24PM (#7233605)
The shootout has been utterly ridiculous for quite a long time. I remember when the maintainer(s?) threw a fit when pypy devs tried getting python versions not relying on cpython tricks (which didn't bench well for pypy) even though multiple programs per language had not been an issue before. Then they ended up banning multiple implementations, but their implementation pick is completely arbitrary.
Re:Rust on Computer Language Benchmarks Game by hansjorg on Thursday February 13, 2014 @03:17PM (#7233981)
The Benchmarks Game maintainer explains why in the comments to this blog post:

http://alexgaynor.net/2011/apr/03/my-experience-computer-lan...

Sounds entirely reasonable to me.

Re:Rust on Computer Language Benchmarks Game by brandonbloom on Thursday February 13, 2014 @02:37PM (#7233697)
Seems like an opportunity for an enterprising hacker to create a more open and collaborative "benchmarking social network" of sorts.
Re:Rust on Computer Language Benchmarks Game by mutagen on Thursday February 13, 2014 @03:01PM (#7233868)
TechEmpower has done pretty well with their web framework benchmarks [1]. While I get the feeling that they don't have the bandwidth to expand into general language benchmarking they've created an open process [2] that might be easily adapted to general language benchmarking.

They seemed to have addressed most of the fairness and "What about language X on platform Y using framework Z" kinds of questions with "Send us a pull request and we'll include it in the next benchmark".

I'm tempted to tackle this even though I don't currently have the time.

[1] http://www.techempower.com/benchmarks/ [2] https://github.com/TechEmpower/FrameworkBenchmarks

Re:Rust on Computer Language Benchmarks Game by pekk on Thursday February 13, 2014 @03:45PM (#7234190)
Except they obstruct issues and pull requests which would fix artificially hamstrung implementations, rather than actively seeking implementations to keep them honest. The benchmarks then represent a fantasy world where completely normal optimizations are disallowed. The point of saying "send us a pull request" is just PR and it seems to be working.
Re:Rust on Computer Language Benchmarks Game by RyanZAG on Thursday February 13, 2014 @03:59PM (#7234283)
AFAIK the only optimization they disallow is the use of caching results. It's clear why they do this: if you allow caching, you're just testing memory fetch speed and all your results across frameworks will probably look about the same. The idea of the benchmark is to actually do the JSON serialization/database access/etc.

I believe they have plans to add additional tests which will allow for caching and that kind of thing.

I don't think that removing caching across all frameworks equally is creating 'artificially hamstrung implementations' as all the frameworks would benefit from caching.

Re:Rust on Computer Language Benchmarks Game by GregBuchholz on Thursday February 13, 2014 @02:54PM (#7233813)
You might think so, but keep in mind that it is a thankless volunteer job that's been on-going for something like 10 years. I'm pretty sure that Isaac would encourage your efforts to make that site.

http://benchmarksgame.alioth.debian.org/play.php#faqs

Re:Rust on Computer Language Benchmarks Game by brandonbloom on Thursday February 13, 2014 @03:13PM (#7233950)
Surely in today's world with elastic compute and robust virtual containers, it should be possible to replace thankless volunteer work with automated, self-service processes.

I'm not saying it would be easy (or economically viable), but it's certainly possible. Somebody should try it.

Re:Rust on Computer Language Benchmarks Game by awj on Thursday February 13, 2014 @03:54PM (#7234251)
Taking the measurements isn't really the thankless volunteer work involved here. The work is in ensuring new programs actually follow the intent of the benchmark and work across supported architectures.

This isn't an automateable task, although it may be a crowdsource-able one. Luckily the benchmark game publishes pretty much all of the tooling they use, so you could probably get a pretty big headstart there on your improvements.

Re:Rust on Computer Language Benchmarks Game by brandonbloom on Thursday February 13, 2014 @04:15PM (#7234403)
Fair enough. Although, I view crowdsourcing as an implementation detail of an automated process.
Re:Rust on Computer Language Benchmarks Game by igouy on Friday February 14, 2014 @11:29AM (#7239349)
>>when pypy devs tried getting python versions not relying on cpython tricks (which didn't bench well for pypy)<<

NOT TRUE.

Programs optimized for PyPy were accepted and shown alongside programs written for CPython. Joe LaFata contributed excellent PyPy pi-digits, spectral-norm, mandelbrot programs and they were all accepted.

>>banning multiple implementations<<

NOT TRUE.

Measurements for both JRuby and CRuby are shown.

Like everyone else, I'm sitting on my hands waiting for someone else to make and publish all the other program measurements.

The implementation pick ranges from the standards that everyone uses to the experimental that nobody but the language developer uses.