Nobody believes Elon anymore.

Hm, impartial benchmarks are independent of Elon's claims?

Impartial benchmarks are great, unless (1) you have so many to choose from that you can game them (which is still true even if the benchmark makers themselves are absolutely beyond reproach), or (2) there's a difference between what you're testing and what you care about.

Goodhart's Law means 2 is approximately always true.

As it happens, we also have a lot of AI benchmarks to choose from.

Unfortunately this means every model basically has a vibe score right now, as the real independent tests are rapidly saturated into the "ooh shiny" region of the graph. Even the people working on e.g. the ARC-AGI benchmark don't think their own test is the last word.

It's also possible they trained on test.

Likely they trained on test. Grok 3 had similarly remarkable benchmark scores but fell flat in real use.

"impartial" how? Do you have the training data, are you auditing to make sure they're not few-shotting the benchmarks?

The latest independent benchmark results consistently output "HEIL HITLER!"

Upvotes are a lagging indicator. Despite all the leaderboard scores presented, etc, no one actually knows how good a model is until they go use it for a while. When Claude 4 got ~2k upvotes, it was because everyone realized that Claude 3.7 was such a good model in practice - it had little to do with the actual performance of 4.

Other AI companies post a 5 minute article to read.

This is a 50 minute long video, many won't bother to watch

I'm not sure there's any benchmark score that'd make me use a model that suddenly starts talking about racist conspiracy theories unprompted. Doubly so for anything intended for production use.

Its a shame this model is performing so well because I can't in good conscience pay money to Elon Musk. Will just have to wait for the other labs to do their thing.

Psychopath.

You misspelled "principles".

Maligning any alternative viewpoints to yours as just some indoctrinated people following “marching orders”, rather than addressing the substance of their critique, constitutes a “poisoning the well” fallacy.

Substance being ?

HN seems to be full of Anthropic fanboys for some reason. Probably because Dario is the only big boss in AI right now that successfully pulls off the I'm not a sociopath act.

Probably more like Claude was slightly better than GPT-xx when the IDE integrations first got widely adopted (and this was also the time where there was another scandal about Altman/OpenAI on the front page of HN every other week) so most programmers preferred Claude, then it got into a virtuous cycle where Claude got the most coding-related user queries and became the better coding model among SOTA models, which resulted in the current situation today.

because grok on twitter has been saying deeply antisemitic stuff as recently as earlier this week. like, how are you supposed to square the circle here: they have the best, most intelligent AI, and it is deeply invested in conspiracy theories created by fascists. what do you do with both of these pieces of information.

You can use a “formula” and make excel write offensive stuff too.

nobody would be claiming an excel spreadsheet is anything close to intelligent tho.

Because the benchmarks are likely gamed. Also Grok had an extremely negative news cycle right before this, so the average bloke is skeptical that the smartest AI in the world thinks the last name Steinberg means someone is a shadowy, evil, cabal-type figure. Even though they aren't totally related, most people aren't deep enough in the weeds to know this