News for hackers. Stuff that matters.
Re:Benchmarks in CI: Escaping the Cloud Chaos by farkin88 on Sunday August 03, 2025 @04:37PM (#44779555)
I've been burnt by performance gates on GitHub Actions. One random timing spike and the whole PR turns red. The coefficient of variation math here nails why: GitHub Actions shows a 2.66% CV, which means a 2% performance gate gives you a 45% false positive rate (basically every other run flags a fake regression). No wonder developers stop trusting the check. In my experience the only way to make benchmarks actionable is to run them on deterministic bare-metal runners, whether CodSpeed's or something you host yourself.

There are no comments on this comment yet.