News for hackers. Stuff that matters.
Re:Qwen4-Max by rhdunn on Monday November 16, 2026 @05:05AM (#50515903)
The key issue with public benchmarks is that they allow the model providers to train against them and artificially inflate their score accordingly. Private benchmarks are a way of making it harder to game the system, though it makes it harder for people looking at the benchmark to assess it as well.

Neither option is perfect.

A third option would be updating the benchmark every 6-12 months, but that has its challenges as well as you need to make sure that the benchmark covers things the model hasn't seen or is harder for the models to game.

Re:Qwen4-Max by Eisenstein on Monday November 16, 2026 @05:22AM (#50516010)
A fourth option is realizing that benchmarks exist because it is impossible to measure the value we want to measure, so we need a stand-in. The stand-in will sometimes correlate with performance but often just act as a way to compare. When used to compare, it is most important that the benchmark be fair and trustworthy.
Re:Qwen4-Max by aenis on Monday November 16, 2026 @05:26AM (#50516030)
Makes me wonder how do you privately benchmark a model hosted by the lab. Even with zdr, the bytes are flowing through their infra and possibly influencing god knows what for the next training round.
Re:Qwen4-Max by nsagent on Monday November 16, 2026 @05:52AM (#50516192)
You use something like item response theory [1] to be able to change the test over time while maintaining validity. It's what standardized tests like the GRE use.

Interestingly, Artificial Analysis does have a score that uses IRT, it's their Progress Index [2].

[1]: https://en.wikipedia.org/wiki/Item_response_theory

[2]: https://artificialanalysis.ai/methodology/progress-index

Re:Qwen4-Max by loehnsberg on Monday November 16, 2026 @08:50AM (#50517589)
Would OpenAI also "see" the request if it went through something like AWS Bedrock? This isn’t their infra and afaik the whole idea of this kind of intermediary.
Re:Qwen4-Max by staticman2 on Monday November 16, 2026 @10:06AM (#50518376)
I thought the major benchmark companies always get advanced access to a model, which presumably is not done via bedrock.
Re:Qwen4-Max by runeblaze on Monday November 16, 2026 @10:25AM (#50518583)
this is an worry I sometimes see. I want to note that these are billion/trillion dollar companies we are talking about that have mutual talent movement. Violating ZDR might be contractual violation, or at the very least, a strong breach of trust.

(not to say there is no value in "zero-trust" environments, but often pure-benchmaxxing is a deadend; some benchmaxxing is always involved and much less risky options than "train on the ZDR test env" usually exist)

Re:Qwen4-Max by irthomasthomas on Monday November 16, 2026 @05:37AM (#50516108)
The private benchmarks can be seen by the labs when you run them. We only have their word that they wont look at them, which isn't worth much. We should assume that everyone benchmaxxes equally, apart from a few outliers, and try to climb the hill by producing relevant training data.
Re:Qwen4-Max by rhdunn on Monday November 16, 2026 @05:51AM (#50516182)
Yes, that's another challenge with constructing benchmarks.

You could run the benchmarks using downstream providers like AWS if they are available.

You could vary the parameters of the tests to both help with LLMs learning as is and to help obscure that this is a test. Especially if sending them through a thirdparty, so the providers don't get identifying markers like IP addresses.

But ultimately it's another arms race like with viruses and anti-virus software.

Re:Qwen4-Max by Topfi on Monday November 16, 2026 @08:33AM (#50517445)
Eh, I have an eval suite dating back to before GPT-5 in parts and have yet to see certain labs do as well on that as the public eval data would suggest.
Re:Qwen4-Max by paimapi on Monday November 16, 2026 @10:06AM (#50518384)
It depends entirely on the financial incentive. Does improving against your eval suite lead to more adoption? Or would focusing on the far more popular benchmarks convince more CTOs to sign a multi-year contract?
Re:Qwen4-Max by Topfi on Monday November 16, 2026 @10:29AM (#50518631)
Who'd sign a multi-year contract based on any benchmarks, public, private, mine or someone elses? We get new frontier LLMs every 0.5-2 months and any CTO of a major entity already uses Bedrock or the like, so why would anyone lock themselves into a provider or even worse a single model for multiple years?
Re:Qwen4-Max by paimapi on Monday November 16, 2026 @12:00PM (#50519831)
Because API usage is extremely expensive and subscriptions are subsidized? We (my company of 100+) just moved off of AWS Bedrock to a locked in plan. Even when I was working for a much larger company (10k+ employees), we had enterprise subscriptions to OpenAI and Anthropic because the token cost for APIs was enormous.
Re:Qwen4-Max by tallesborges92 on Monday November 16, 2026 @07:00AM (#50516632)
Livebench does it
Re:Qwen4-Max by Uke on Monday November 16, 2026 @07:12AM (#50516742)
Or to separate the benchmark data by type and enable the user to filter by private, public or both.
Re:Qwen4-Max by skohan on Monday November 16, 2026 @08:13AM (#50517275)
I think the best solution is really to update the benchmark periodically, and publish the benchmarks after the results are published.

Otherwise you have to rely completely on trust that AA is really measuring intelligence.

Re:Qwen4-Max by bee_rider on Monday November 16, 2026 @10:26AM (#50518599)
It’s also worth keeping in mind the other alternative, having internal benchmarks that more accurately reflect your workload, that you know all about, and the LLM provider doesn’t. There doesn’t seem to be a way to maintain such a thing specifically for the purposes of online chatter (both because it isn’t profitable, and because things like “your workload” aren’t possible to define in that case). So we should be happy with what we’ve got.

This is a permanent problem in benchmarking really. Transparency is required for trust but invites over-tuning.

Re:Qwen4-Max by rhdunn on Monday November 16, 2026 @10:42AM (#50518800)
There are tools like promptfoo that can be used for internal benchmarks and evaluations that allow you to compare results across different models and runs. This can be useful for personalized benchmarks and for business applications where you need to evaluate models and/or system prompts.
Re:Qwen4-Max by satvikpendem on Monday November 16, 2026 @12:27PM (#50520175)
Why wouldn't the LLM provider know about your internal benchmarks when they use prompts and chats as training data? They can easily learn about your benchmarks too.
Re:Qwen4-Max by MetalBreaksAndB on Monday November 16, 2026 @01:00PM (#50520636)
Just use ZDR.