If someone who didn't know Lua wrote those programs, then you'd complain that the programs weren't written by an expert.
If Lua wasn't included then I think you'd complain about that too.
I happened to spot a problem on this one on the Lua part and also on some other parts concerning how they classify things, because it happens to be things I know more. I'm not going out of my way to look for problems in the rest of the paper because I'm not a reviewer and I have better things to do.
But if I was the one way conducting this academic research, then I would want the results not to be bogus. The difference here is this is not a blog post about some bullshit comparison someone is making. If that was the case, then alright, if public contributed algorithms failed to represent exactly what was being looked for, why not go for whatever is out there. I just expect a more rigorous procedure from academics.
Are they likely to be expert in all those programming languages?
----
If Lua wasn't included would you complain ?
( Like this:
https://news.ycombinator.com/item?id=15255427
https://news.ycombinator.com/item?id=15251242
https://news.ycombinator.com/item?id=15251144
)
No, but again, I take paper results very seriously. Misleading results are bad for science, period. Here is an example of more rigorous research: https://arxiv.org/pdf/1602.00602.pdf
> If Lua wasn't included would you complain ?
Maybe I would, who knows. I did also mention in another comment it would be nice if Julia was there. It depends how relevant they are for the research being conducted. As a language used very often in microcontrollers, Lua is very relevant for energy efficiency research. I don't know anything about Forth, but I wouldn't just dismiss people saying it should be there with "then contribute forth algorithms to the benchmarks game yourself".
"disgusts me" & "pure dishonesty" don't seem like a serious response.
> … an example of more rigorous research…
Which again uses programs that were contributed to the benchmarks game.
> … wouldn't just dismiss people…
I wouldn't just dismiss them: I'd tell them that others have every right to present what seems important to them, and exclude what seems less important -- without being accused of dishonesty.
There's only so-much time & money.
No mystery: someone has made the effort to contribute all those multi-process Python programs; no one has made the effort to contribute all those multi-process Lua programs.