Neither option is perfect.
A third option would be updating the benchmark every 6-12 months, but that has its challenges as well as you need to make sure that the benchmark covers things the model hasn't seen or is harder for the models to game.
Interestingly, Artificial Analysis does have a score that uses IRT, it's their Progress Index [2].
[1]: https://en.wikipedia.org/wiki/Item_response_theory
[2]: https://artificialanalysis.ai/methodology/progress-index
(not to say there is no value in "zero-trust" environments, but often pure-benchmaxxing is a deadend; some benchmaxxing is always involved and much less risky options than "train on the ZDR test env" usually exist)
You could run the benchmarks using downstream providers like AWS if they are available.
You could vary the parameters of the tests to both help with LLMs learning as is and to help obscure that this is a test. Especially if sending them through a thirdparty, so the providers don't get identifying markers like IP addresses.
But ultimately it's another arms race like with viruses and anti-virus software.
Otherwise you have to rely completely on trust that AA is really measuring intelligence.
This is a permanent problem in benchmarking really. Transparency is required for trust but invites over-tuning.
I have been using Terra a lot and I'd say that most of the models (even small ones) are are roughly equivalent when it comes to working in a consistent, established codebase.
But in practice, Terra is not very good at staying on task compared to say Sol 6.1 or Sonnet 5.5. Terra 6.1 will often just pause, thinking the objective has been met or the turn has ended.
I can't remember the last time openAI was at the top of the leaderboard?
Google had a brief moment at the top recently but other than that I feel like it's always Anthropic.
For most things I'd use Kimi is probably better so the ranking is wrong for me.
However it also seems like in those benchmarks where Kimi underperformed it did do quite badly, whereas when Astra underperformed it did so less badly. How you reflect that into a single number I'm not sure - it's certainly difficult to get right and it doesn't seem obviously biased to me.
Certain labs have been shown to and will continue to train on public evals, whether unintentional or on purpose, any respectable ranking across multiple evals also needs private data to counteract this.
Do you know a better site?
You can easily come up with a difficult private benchmark and test them yourself to validate the intelligence gap. It is real.
China lost. There is no catching up. Ever. That ship sailed.
What’s the evidence for this? Are there plausible alternate explanations?