News for hackers. Stuff that matters.

A comment on Claude Opus 4.1.

Re:Claude Opus 4.1 by alrocar on Tuesday August 05, 2025 @01:32PM (#44801273)
just ran the LLM to SQL benchmark over opus-4.1 and it didn't top previous version :thinking: => https://llm-benchmark.tinybird.live/
[ Parent | on HN ]
Re:Claude Opus 4.1 by epolanski on Tuesday August 05, 2025 @01:38PM (#44801377)
How does running it multiple times performs?

LLMs are non-deterministic, I think benchmarks should be more about averages of N runs, rather than single shot experiments.

[ Parent | on HN ]