Show HN: Hyperfine – a command-line benchmarking tool

Re:Show HN: Hyperfine – a command-line benchmarking tool by kqr on Tuesday October 15, 2019 @05:33AM (#21256868)

I don't fully agree. Imagine there are two programs: Acme and Blob. Acme theoretically runs faster, but it also degrades much worse under interference from other system loads. Blob doesn't run as fast at the theoretical maximum, but it handles system load very well and doesn't degrade too bad.

There's an argument to Blob being the better choice, if this will run in production on a system that might encounter unrelated loads. Predictable performance is frequently more useful than theoretical maximum performance.

That said, I still agree a little bit. The minimum value is also a useful metric, and if you have the opportunity to report two numbers, the minimum-maximum pair is a great choice.

[ Parent | on HN ]

Re:Show HN: Hyperfine – a command-line benchmarking tool by bradknowles on Tuesday October 15, 2019 @05:41PM (#21264175)

I disagree. I think you want to report as much as you reasonably can about the distribution, and then you have to let the reader make up their mind which they think is most important.

You can have your own idea about which is most important and build your comparisons on that choice, but you should give the reader enough information that the work you put in is still valuable to them in case they should make a different choice.

If we both have equal information about the distribution, then we can make our own choices about the data being presented.

[ Parent | on HN ]