News for hackers. Stuff that matters.
Re:OTelBench: AI struggles with simple SRE tasks (Opus 4.5 s... by winton on Thursday January 29, 2026 @11:23AM (#46812344)
So if I try to do it with Opus three or four times, I'll get it done? And probably in about 10 minutes? Awesome
Re:OTelBench: AI struggles with simple SRE tasks (Opus 4.5 s... by stared on Thursday January 29, 2026 @11:45AM (#46812644)
Nope, these are no random dice rolls. Some times are solved each run, a few - occasionally (so here would be meaningful to try a few times - and metrics of pass@1 and pass@3 would be different), but most are never solved.

See e.g.: https://quesma.com/benchmarks/otel/models/claude-opus-4.5/

Re:OTelBench: AI struggles with simple SRE tasks (Opus 4.5 s... by throwup238 on Thursday January 29, 2026 @11:29AM (#46812435)
That’s only if the failures are truly random and aren’t correlated