It is not very good at hard tasks, its ranking is much worse there.

sorry, any examples of hard tasks

I used to defend LMSys/Chatbot Arena a lot but threw in the towel after events of the past three months.

I can give more details if you (or anyone else!) is interested.

TL;DR: it is scoring only for "How authoritative did the answer look? How much flattering & emojis?"

Is this not what Style Control (which IIRC they're making default soon) aims to mitigate?

I'm not 100% sure what their rationale is for it, the launch version of style control was a statistical model that penalized a few (4?) markdown shibboleths (lists, headers, ?).

Not sure if they've shared more since.

IMVHO it won't help, at all, even if they trained a perfect model that could accurately penalize it*

The main problem is its one off responses, A/B tested. There's no way to connect it into all the stuff we're using to do work these days (i.e. tools / MCP servers), so at this point its sort of skipping the hard problems we'd want to see graded.

(this situation is a example: whats more likely, style control is a small idea for an intractable problem, or Google has now released multiple free models better than Sonnet, including the latest, only 4B params?

To my frustration, I have to go and bench these things myself because I have an AI-agnostic app I build, but I can confirm it is not the case that Gemma 3-not-n is better than Sonnet. 12B can half-consistently make file edits, which is a major step forward for local tbh)

* I'm not sure how, "correctness" is a confounding metric here: we're probably much more likely to describe a formatted answer in negative terms if the answer is incorrect.

In this case I am also setting aside how that could be done, just saying it as an illustration of no matter what, it's the wrong platform for a "how intelligent is this model?" signal, at this point, post-Eliza post-Turing, couple years out from ChatGPT 1.0

The catch is that "does as good as X" is pretty much never representative of real world performance when it comes to LLMs.

In general, all those scores are mostly useful to filter out the models that are blatantly and obviously bad. But to determine whether the model is actually good at any specific thing that you need, you'll have to evaluate them yourself to find out.