Digging into details and yielding very specific conclusions.

Let’s take the example benchmarks.

The first one, Fibonacci, can only be properly assessed by inspecting the assembly. It is an analysis of compiler output, not of the language, not even of compiler optimizations.

It therefore only measures the CPU speed of function calls, and one compiler outputs a different set of assembly instructions:

• Go (go tool objdump fib) is all like “CMPQ JBE(label(morestack)) … LEAL(-1,cx) MOVL(cx) CALL(fib) …×3 ADDL(-2,cx) MOVL(cx) CALL(fib) MOVL ADDL …×3 RET label(morestack) CALL(morestack) JMP”.

• Crystal (crystal build --emit=asm fib.cr) is all like “subl(1) …×6 callq(fib) …×1 subl(2) …×7 callq(fib) movl addl …×6 addq retq” for the recursive condition.

So the conclusion is that Go outputs less instructions, but has to maintain its segmented stacks (useful for goroutines, its lightweight threading system) on every call, while Crystal uses the standard assembly calling system with little overhead beyond light dynamic typechecks.

So there is a language-motivated difference: calls in Go are a tiny bit more expensive, but built-in lightweight threads consume little memory because their stack can increase dynamically according to use[0].

The second benchmark, HTTP servers, has everything to do with the implementation of the default library, and nothing to do with either the language or the compiler. And I presume Crystal is backed by an optimized C library for TCP[1] while Go is probably a full reimplementation of TCP and HTTP.

[0]: https://blog.cloudflare.com/how-stacks-are-handled-in-go/

[1]: https://github.com/crystal-lang/crystal/blob/master/src/sock...

I think they dropped segmented stacks in 2014: https://en.wikipedia.org/wiki/Go_(programming_language)#Vers...

The stack can still grow, but contiguously. I'm not sure if there are any instructions required to monitor the stack size and grow if it needed.

^^ It's still the preamble for most functions :)

On what basis are you presuming that?

Obviously there are an unbounded set of ways people can test the boundaries or contribute implementations that aren't in the spirit of the project. We've had to deal with several contributions that required push back or removal. Since we run our benchmarks project in our spare time, we rely heavily on the community to help us keep an eye on implementations, and also help determine where to set boundaries when they require clarification. For example, is pipelining in the Postgres driver acceptable given we disallowed batch querying? We ultimately decided it was acceptable because it's leaving the ergonomics of application development unchanged.

I feel that overall, the project has been effective in large part because the community has done a good job of self-policing.

[1] https://www.techempower.com/benchmarks/

I've never seen a situation where a team genuinely looks at the available languages for their project and chooses one based purely on performance. Even projects that are very performance-sensitive usually have other criteria (like not being able to use a GC language).

In most situations, there are a number of competing criteria for choosing a language. Team familiarity, deployment environment, availability of necessary libraries, compatability with other code in the organisation, compilation time, making maximum use of available resources. All of these, and many more, are usually as important as raw speed.

For a "real" comparison, there would probably need to be a lot more code being executed, and specific tests for specific environments - standalone web server? microservice? CLI? desktop app? mobile app? All of them will have different needs and preferences for performance.

There are different projects that allow people to compare languages in real scenarios:

- Good old TodoMVC (http://todomvc.com/)

- HNPWA (https://hnpwa.com/), that compares PWAs for accessing the HN api

- RealWorld (https://github.com/gothinkster/realworld) that does the same but for both frontend and backend. The API between both is standardized, so you must be able to mix and match any frontend (including mobile) with any backend. The idea is to build a Medium clone, so it's much closer to a real use case than a todo list

More than the language, the framework and the architecture can play a role in speed. If I cared enough about relative performance, I'd build a client that adheres to the API spec of RealWorld and simulate a few thousand clients on a few thousands posts, all reading and writing posts and comments, and see how well they compare under load. That would be much more telling than comparing who can produce the nth Fibonacci number the fastest.

When "idiomatic" performs so poorly that the software is unfit for purpose, does that represent "real code" ?

> … solutions to the same problem from many different programmers at different skill levels…

How could we know programs weren't written to make some language look bad?

I'm confused about what argument you're making here. I assume you aren't suggesting that suboptimal performance on a benchmark implies that the software is unfit for purpose? Can you elaborate?

Hence "idiomatic" may represent not "real code" but "naive code".

Do you intend to provide some support for your idea that "it's useful to know about the performance of idiomatic software"?

No, they're dismissing it as useful for answering the questions we tend to care about when evaluating the performance of various programming languages. One such question is "If I choose a given language, how far will I be able to go with just idiomatic code before I need to invest in expensive optimizations (most of the expense here is that we're spending time optimizing instead of iterating on features)?". "Real code" is a function of this concern, so measuring it doesn't help us answer this question (i.e., if a language performs poorly on this question, "real code" will be more optimized and less idiomatic). Further, microbenchmarks don't measure "real code" either they maximally optimized code, which is what the OP was complaining about.

Your question is sensibly answered by — It depends.

When you're actually aiming for performance, in my experience code stops looking like "real code" pretty quickly in the hotspot, so benchmarks are representative in that way.

I mean, for example media codecs "written in C" are typically highly vectorised, architecture-specific ASM kernels with C around it for handling the flow.

The most fair benchmarks I've seen are to implement strictly numerical algorithms, but that's not very userful because you're basically testing a small piece of compiler/VM optimization at that point. It also leads to unidiomatic code because it's faster, e.g. "C in Java".

The most useful benchmarks I've seen are exercises where programmers of different languages try their best to create as fast a program in their language as possible. It allows you to see idiomatic v.s. fast code in that language. Downside is you're testing for a large part which contributors you've managed to catch.

Why would that be the case? I see no correlation. The two programs in the article were little more than glorified "hello, world" samples.

For all we know, Crystal might accumulate binary code cruft much quicker than Go as the project grows.

And more: Them are the MOST useful comparisons. Compare 2 oranges is not as useful than compare an apple and orange. You have MORE useful differences and more insights on what make them similar.

In the case of speed:

I compare a bike and a car. Is unfair? yes. But, if the GOAL is how fast is go to the next city, is totally worth to know that the car is better.

However, if the goal is what is the most ENVIRONMENT FRIENDLY why to travel, the bike win.

Comparing 2 very different vehicles is far more useful.

---

What is important in benchmarks? To make clear WHAT is the goal to be measured. And not shy away to hammer if some choice become clearly superior.

The problem with these benchmarks is they have a propensity to not make a fair comparison. For example, sometimes they try to compare underperforming naive implementations in one language with expertly tuned implementations in another. Sometimes a blatantly underperforming implementation is used. Sometimes results are turned upside down by flipping compiler flags.

And sometimes photofinish results that fall within the standard deviation are sold as this absolute proof that language A outperforms language B.

Let's look at these benchmarks to see how these things are spun to sound like objective assertions are derived from meaningless data. Does it really matter if someone could use language A to shave off about 1MB from build artifacts? Are we really expected to take a 1.5s/5% difference in a cherry picked synthetic benchmark as meaningful? Is anyone in their right mind expected to drive their decisions on their tech stack on two cherry-picked tricks posted on a blog?

This is why no one takes these stunts seriously.

The corollary is to write your own benchmark (i.e. something that remotely matches your use case).

Also it is almost always more interesting why something is slower/faster than something else than the raw numbers.

Sure. But if that code is going to be ran a lot, please try to keep the number of required nuclear power plants down.

Also don't forget about phone and laptop battery life either.

To use your analogy, I'd saying comparing two languages is more like comparing a woodworking workshop to a metal working workshop. There's a lot of similarities (they both have hammers) and there's a lot of overlap in functionality (you could make a little box in both workshops). But how do you make a sensible comparison between the two shops? Just saying 'you can build this little box slightly faster in the woodworking shop' doesn't really tell you much at all. They're tools that can be applied to a wide range of tasks, and unless you have a specific task you're interested in (e.g. I want to be able to process my log files as quickly as possible) it's hard to make any general statement about one set of tools being better or worse than another.

The old 0install “language replacement” shootout is definitely the one I remember best.

It’s really outdated now (and oddly enough not always for the better) but:

* it evaluated multiple dimensions / criteria

* the criteria were very explicitly about relevance to the tester

* but their testing was quite objective

* and it really was quite extensive

The problem is that yes, they are an apples to oranges comparison. You may use them for the same problems, but different languages lead to different styles, different trade-offs, and even different ways of thinking.

As an extreme example, a slow language may make it easier to do some high-level optimization that you would never be able to do on a fast language, getting a much faster program. And we have no reliable way to measure or communicate that kind of thing.

This of course still misses how long it took to implement, how easily changes can be made, etc.

1 - http://todomvc.com/

With genuine interest and curiosity.

With the understanding that any conclusions will be provisional.

With incredible focus on the context that matters to you.

https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

The results do say something about golang and crystal. Without more data we cannot confirm anything, therefore the following possibility remains an open question:

Crystal may be generally faster and more efficient than golang for most use cases.

To not acknowledge this possibility or to fully embrace it as an absolute is a form of bias.

Did you actually see programs that do not follow the guidelines or you're just stating this as what you would expect?

Templating Lua here is probably borderline really.