I have no context regarding the history of the benchmarks game. With that in mind, I have a question:

What about you create a technical specification (including both technical depth and common-sense breadth) for what kind of benchmark you'll accept, throw the whole thing on GitHub, and then refuse 99% of pull requests? :) (ie, only accept really really good quality benchmark implementations)

Eventually, enough developers unimpressed that Language X is not adequately represented would step up to the plate and maintain good-quality benchmark code. (This could get pretty interesting with rapidly-evolving languages like Rust.)

Obviously this is all very ideal and I can see so many ways such an endeavor could go horribly wrong, sure. I can also very easily see you having floated such an idea then discarded it for reasons I haven't even thought of.

What about I work to publish crowd-sourced programs and measurement scripts: so that anyone can make their own comparisons, on their own hardware, against whatever other language implementations they want to write programs for :-)

https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

Ah; that certainly works too!

The only thing I wonder about is allowing any given set of hardware to be compared with another set of hardware with anything approaching accuracy.

What would be awesome is if someone could figure out whether they'd need Raspberry Pi Zero to run some algorithm or if they could get away with just using a BBC Micro:Bit, by looking at a benchmark run submitted to the site from an i9-7940X.

Sadly I suspect the amount of realtime introspection needed (memory speed, practical cache coherency (per level), bus saturation, instructions used per cycle) would make this difficult - particularly because, even if a given benchmark was going to go fishing for performance counter info (and I just learned that even the Gen1's ARM 1176-based PMU provides a few, including one that counts instructions), benchmarking memory I/O is a bit harder; PCI DMA MMIO debug cards only map a small range of memory, like 256k or so, and I don't know if such cards can back the region with actual RAM. I suspect the access latency would be so different than from normal RAM that even if this did exist it'd never be used. Sigh.

And then differences in compiler optimization approach would have to be taken into account, and thorough understanding of assembly language for all target architecture(s) would be needed to have the time of day to page through the diff analysis...

Hmm, I think "simple" got left behind a few thousand miles ago. It's kinda (morbidly?) fascinating how different that different architectures are on the ground, hah.

Oops:

- To clarify, I meant the Gen1 Raspi.

- I now realize (after having actually had a proper look) that while the Micro:Bit does use "JavaScript" and "Python", they're different, microcontroller-specific implementations, and I should have used a different example there, like an Intel NUC or perhaps an ODROID board or similar.

>> The only thing I wonder about is allowing any given set of hardware to be compared with another set of hardware with anything approaching accuracy.

Just systematically enough to be reminded that a different context may have different relative performance.

You've never explained what your criteria is. Just that you didn't want to include D.

I always did want to include D — which is why we did include D — of course, we also used to include Scala and Clojure and…

I know you're publicly directly messaging, but as an outsider to the discussion, I don't see any actual mention in your posts why D isn't included. Just that it "used to be."

There are many of benchmark projects that include D and don't appear to be struggling under any kind of massive burden. For example:

https://github.com/kostya/benchmarks

What are your actual reasons for not keeping D... "scala... and Clojure and ..."? The results in the previous link show D as a massive competitor (sometimes 1st place beating C and C++) on both memory and speed. Wouldn't the purpose of benchmarks be... to highlight useful, highly-scoring languages? Isn't that one of the primary reasons people read benchmarks?

(The D implementations are also often smaller in lines of code, as per this benchmark project:

https://togototo.wordpress.com/2013/08/23/benchmarks-round-t...

)

>> I don't see any actual mention in your posts why D isn't included.

Do you see any mention at kostya/benchmarks of why ".NET Core" or Ada isn't included?

>> … benchmark projects that include D and don't appear to be struggling under any kind of massive burden.

Check when those kostya/benchmarks language implementation versions were released.

>> What are your actual reasons for not keeping…

I don't want to do the work.