News for hackers. Stuff that matters.
Re:Benchmarking 20 programming languages on N-queens and mat... by Atotalnoob on Wednesday January 03, 2024 @12:38AM (#38851120)
Aren’t JIT languages at a disadvantage since they are benchmarked through the CLI rather than using a benchmarking library to allow JIT to warmup?
Re:Benchmarking 20 programming languages on N-queens and mat... by brabel on Wednesday January 03, 2024 @02:46AM (#38851703)
For one-time problem runs, the JIT languages in practice will not be given time to warm up. All that matters for a user is how fast the application is in practice. It's not about "making it fair" for languages, it's about measuring how fast they go from nothing to results.

It doesn't make sense to allow "warmup" time for them unless your expected application is a server which for most of the time will be running "warm" (and even then with scalable containers that assumption may not even be true in some cases). For servers, however, what matters is mostly how fast its HTTP library is and how good the async IO is... check Techempower benchmarks for that: https://www.techempower.com/benchmarks/#hw=ph&test=fortune&s...

Re:Benchmarking 20 programming languages on N-queens and mat... by nithril on Wednesday January 03, 2024 @12:02PM (#38856449)
From nothing to result rarely happens in real live. I hardly see someone to start/stop a program per unit task (like piping commands).
Re:Benchmarking 20 programming languages on N-queens and mat... by jltsiren on Wednesday January 03, 2024 @01:04PM (#38857345)
Code compilation is a common example. In many languages, when you do a clean build of a large project, you run a large number of short-lived processes.

With command line tools, the dominant design philosophy is that a program completes one task and exits. The tasks may be large or small, and the same tools are often expected to handle both.

Re:Benchmarking 20 programming languages on N-queens and mat... by jakobnissen on Wednesday January 03, 2024 @01:25AM (#38851353)
Yes, but the author claims the longest JIT warmup is 0.3 seconds, so it's not an important issue in these benchmarks that take several seconds.
Re:Benchmarking 20 programming languages on N-queens and mat... by lifthrasiir on Wednesday January 03, 2024 @01:32AM (#38851378)
I strongly suspect that the author may have confused the JIT warmup (hard to measure, as you need to ensure that the performance figure have reached the stable point) from the startup overhead (easy to measure).
Re:Benchmarking 20 programming languages on N-queens and mat... by vkazanov on Wednesday January 03, 2024 @02:29AM (#38851620)
...and many jitted vms do not even reach a stable point at all, there was a big paper on this a couple of years ago.
Re:Benchmarking 20 programming languages on N-queens and mat... by igouy on Wednesday January 03, 2024 @02:38PM (#38858743)
Re:Benchmarking 20 programming languages on N-queens and mat... by vkazanov on Wednesday January 03, 2024 @04:14PM (#38860027)
Yes, this is the one. Time flies! And VM jits didn't change all that much.
Re:Benchmarking 20 programming languages on N-queens and mat... by nithril on Wednesday January 03, 2024 @11:55AM (#38856348)
0.3s is significant for a task that only takes 1.14s
Re:Benchmarking 20 programming languages on N-queens and mat... by igouy on Wednesday January 03, 2024 @06:10PM (#38861068)
We don't know that OpenJDK was "Some JIT-based language runtimes take up to ~0.3 second to compile and warm-up"

Maybe that was PyPy with program times >5 seconds.

Re:Benchmarking 20 programming languages on N-queens and mat... by borissk on Wednesday January 03, 2024 @01:18AM (#38851305)
What's CLI?
Re:Benchmarking 20 programming languages on N-queens and mat... by zogrodea on Wednesday January 03, 2024 @02:41AM (#38851674)
Presumably command line interface/terminal. I can type `dotnet run` in the terminal and provide some options to runa .NET program for example.
Re:Benchmarking 20 programming languages on N-queens and mat... by igouy on Thursday January 04, 2024 @05:35PM (#38873243)
Some language implementations have run time startup costs (bytecode verifier), not-required at run time by other language implementations.

https://www.oracle.com/java/technologies/security-in-java.ht...

Why should that run time startup cost be ignored?