News for hackers. Stuff that matters.
Re:Qwen4-Max by aenis on Monday November 16, 2026 @05:26AM (#50516030)
Makes me wonder how do you privately benchmark a model hosted by the lab. Even with zdr, the bytes are flowing through their infra and possibly influencing god knows what for the next training round.
Re:Qwen4-Max by nsagent on Monday November 16, 2026 @05:52AM (#50516192)
You use something like item response theory [1] to be able to change the test over time while maintaining validity. It's what standardized tests like the GRE use.

Interestingly, Artificial Analysis does have a score that uses IRT, it's their Progress Index [2].

[1]: https://en.wikipedia.org/wiki/Item_response_theory

[2]: https://artificialanalysis.ai/methodology/progress-index

Re:Qwen4-Max by loehnsberg on Monday November 16, 2026 @08:50AM (#50517589)
Would OpenAI also "see" the request if it went through something like AWS Bedrock? This isn’t their infra and afaik the whole idea of this kind of intermediary.
Re:Qwen4-Max by staticman2 on Monday November 16, 2026 @10:06AM (#50518376)
I thought the major benchmark companies always get advanced access to a model, which presumably is not done via bedrock.
Re:Qwen4-Max by runeblaze on Monday November 16, 2026 @10:25AM (#50518583)
this is an worry I sometimes see. I want to note that these are billion/trillion dollar companies we are talking about that have mutual talent movement. Violating ZDR might be contractual violation, or at the very least, a strong breach of trust.

(not to say there is no value in "zero-trust" environments, but often pure-benchmaxxing is a deadend; some benchmaxxing is always involved and much less risky options than "train on the ZDR test env" usually exist)