Re:Claude Code Daily Benchmarks for Degradation Tracking
by account266928 on Thursday January 29, 2026 @02:59PM
(#46815728)
Please try to make this statistically rigorous. There's lots of advice in this thread (intraday variation, etc) but if Im reading this right it looks like the CI includes the baseline value yet you still label this as failing.
Wouldn't this just be "our test isn't powerful enough to find a signal if there were one here?"
People will see this and derive strong conclusions that the data don't support and you, `qwesr123`, or "JB" from your blogs, will be responsible.
Wouldn't this just be "our test isn't powerful enough to find a signal if there were one here?"
People will see this and derive strong conclusions that the data don't support and you, `qwesr123`, or "JB" from your blogs, will be responsible.