News for hackers. Stuff that matters.
Re:CodSpeed: Continuous Performance Measurement by jeremiec on Tuesday July 11, 2023 @11:47AM (#36682668)
> And, to eliminate the usual variance associated with running them, we measure the number of instructions and memory/cache accesses through CPU instrumentation performed with Valgrind. This approach gives repeatable and consistent results that couldn’t be obtained with a time based statistical approach, especially in extremely noisy CI and cloud environments.

Impressive! How repeatable are the result with this strategy? Repeatable enough to raise warning on a 10% performance slowdown for example?

Re:CodSpeed: Continuous Performance Measurement by art049 on Tuesday July 11, 2023 @12:19PM (#36683193)
Yes, totally. Most of the time we can even raise a warning with an even lower regression threshold, for example in this benchmark from pydantic-core, there is a less than 1% variance in the results and we can easily detect performance changes of less than 5%: https://codspeed.io/pydantic/pydantic-core/benchmarks/tests/...