Re:Qwen4-Max
by aenis on Monday November 16, 2026 @05:26AM
(#50516030)
Makes me wonder how do you privately benchmark a model hosted by the lab. Even with zdr, the bytes are flowing through their infra and possibly influencing god knows what for the next training round.
[ Parent
| on HN ]
You use something like item response theory [1] to be able to change the test over time while maintaining validity. It's what standardized tests like the GRE use.
Interestingly, Artificial Analysis does have a score that uses IRT, it's their Progress Index [2].
[1]: https://en.wikipedia.org/wiki/Item_response_theory
[2]: https://artificialanalysis.ai/methodology/progress-index
Re:Qwen4-Max
by loehnsberg on Monday November 16, 2026 @08:50AM
(#50517589)
Would OpenAI also "see" the request if it went through something like AWS Bedrock? This isn’t their infra and afaik the whole idea of this kind of intermediary.
Re:Qwen4-Max
by staticman2 on Monday November 16, 2026 @10:06AM
(#50518376)
I thought the major benchmark companies always get advanced access to a model, which presumably is not done via bedrock.
this is an worry I sometimes see. I want to note that these are billion/trillion dollar companies we are talking about that have mutual talent movement. Violating ZDR might be contractual violation, or at the very least, a strong breach of trust.
(not to say there is no value in "zero-trust" environments, but often pure-benchmaxxing is a deadend; some benchmaxxing is always involved and much less risky options than "train on the ZDR test env" usually exist)