News for hackers. Stuff that matters.
Re:Hyperfine: A command-line benchmarking tool by anotherhue on Tuesday November 19, 2024 @01:50AM (#42180660)
It spawns a new process each time right? I would think that would but a cap on how accurate it can get.

For my purposes I use it all the time though, quick and easy sanity-check.

Re:Hyperfine: A command-line benchmarking tool by oguz-ismail on Tuesday November 19, 2024 @02:09AM (#42180749)
It spawns a new shell for each run and subtracts the average shell startup time from final results. Too much noise
Re:Hyperfine: A command-line benchmarking tool by PhilipRoman on Tuesday November 19, 2024 @02:40AM (#42180880)
The shell can be disabled, leaving just fork+exec
Re:Hyperfine: A command-line benchmarking tool by sharkdp on Tuesday November 19, 2024 @05:48AM (#42182044)
Yes. If you don't make use of shell builtins/syntax, you can use hyperfine's `--shell=none`/`-N` option to disable the intermediate shell.
Re:Hyperfine: A command-line benchmarking tool by oguz-ismail on Tuesday November 19, 2024 @07:19AM (#42182636)
You still need to quote the command though. `hyperfine -N ls "$dir"' won't work, you need `hyperfine -N "ls ${dir@Q}"' or something. It'd be better if you could specify commands like with `find -exec'.
Re:Hyperfine: A command-line benchmarking tool by PhilipRoman on Tuesday November 19, 2024 @07:32AM (#42182728)
Oh that sucks, I really hate when programs impose useless shell parsing instead of letting the user give an argument vector natively.
Re:Hyperfine: A command-line benchmarking tool by sharkdp on Tuesday November 19, 2024 @08:20AM (#42183105)
I don't think it's useless. You can use hyperfine to run multiple benchmarks at the same time, to get a comparison between multiple tools. So if you want it to work without quotes, you need to (1) come up with a way to separate commands and (2) come up with a way to distinguish hyperfine arguments from command arguments. It's doable, but it's also not a great UX if you have to write something like

    hyperfine -N -- ls "$dir" \; my_ls "$dir"
Re:Hyperfine: A command-line benchmarking tool by oguz-ismail on Tuesday November 19, 2024 @08:53AM (#42183430)
> not a great UX

Looks fine to me. Obviously it's too late to undo that mistake, but a new flag to enable new behavior wouldn't hurt anyone.

Re:Hyperfine: A command-line benchmarking tool by forrestthewoods on Tuesday November 19, 2024 @02:03AM (#42180722)
The issue is it runs a kajillion tests to try and be “statistical”. But there’s no good way to say “just run it for 5 seconds and give me the best answer you can”. It’s very much designed for nanosecond to low microsecond benchmarks. Trying to fight this is trying to smash a square peg through a round hole.
Re:Hyperfine: A command-line benchmarking tool by PhilipRoman on Tuesday November 19, 2024 @02:42AM (#42180891)
I disagree that it is designed for nano/micro benchmarks. If you want that level of detail, you need to stay within a single process, pinned to a core which is isolated from scheduler. At least I found it almost impossible to benchmark assembly routines with it.
Re:Hyperfine: A command-line benchmarking tool by gforce_de on Tuesday November 19, 2024 @02:39AM (#42180876)
At least it gives some numbers and point in a direction:

  $ hyperfine --warmup 3 './hello-world-bin-sh.sh' './hello-world-env-python3.py'
  Benchmark 1: ./hello-world-bin-sh.sh
    Time (mean ± σ):       1.3 ms ±   0.4 ms    [User: 1.0 ms, System: 0.5 ms]
  ...
  Benchmark 2: ./hello-world-env-python3.py
    Time (mean ± σ):      43.1 ms ±   1.4 ms    [User: 33.6 ms, System: 8.4 ms]
  ...
Re:Hyperfine: A command-line benchmarking tool by sharkdp on Tuesday November 19, 2024 @06:00AM (#42182129)
> The issue is it runs a kajillion tests to try and be “statistical”.

If you see any reason for putting “statistical” in quotes, please let us know. hyperfine does not run a lot of tests, but it does try to find outliers in your measurements. This is really valuable in some cases. For example: we can detect when the first run of your program takes much longer than the rest of the runs. We can then show you a warning to let you know that you probably want to either use some warmup runs, or a "--prepare" command to clean (OS) caches if you want a cold-cache benchmark.

> But there’s no good way to say “just run it for 5 seconds and give me the best answer you can”.

What is the "best answer you can"?

> It’s very much designed for nanosecond to low microsecond benchmarks.

Absolutely not. With hyperfine, you can not measure execution times in the "low microsecond" range, let alone nanosecond range. See also my other comment.