News for hackers. Stuff that matters.
Re:AI benchmarks are a bad joke – and LLM makers are the one... by ACCount37 on Saturday November 08, 2025 @09:59AM (#45857058)
Ratings on LMArena are too easily gamed.

Even professional human evaluators are quite vulnerable to sycophancy and overconfident-and-wrong answers. And LMArena evaluators aren't professionals.

A lot of the sycophancy mess that seeps from this generation of LLM stems from reckless tuning based on human feedback. Tuning for good LMArena performance has similar effects - and not at all by a coincidence.

Re:AI benchmarks are a bad joke – and LLM makers are the one... by energy123 on Saturday November 08, 2025 @11:34AM (#45857932)
It's biased to small context performance, which is why I don't pay much attention to it as a developer aside from a quick glance. I need performance at 40-100k tokens which models like Deepseek can't deliver but Gemini 2.5 Pro and ChatGPT 5.0 Thinking can.
Re:AI benchmarks are a bad joke – and LLM makers are the one... by ACCount37 on Saturday November 08, 2025 @12:17PM (#45858265)
And even "long term performance" splits itself into "performance on multi-turn instruction following" and "performance on agentic tasks" down the line. And "performance on agentic tasks" is a hydra in itself.

Capturing LLM performance with a single metric is a hopeless task. But even a single flawed metric beats no metrics at all.