Re:Qwen4-Max
by irthomasthomas on Monday November 16, 2026 @05:37AM
(#50516108)
The private benchmarks can be seen by the labs when you run them. We only have their word that they wont look at them, which isn't worth much. We should assume that everyone benchmaxxes equally, apart from a few outliers, and try to climb the hill by producing relevant training data.
[ Parent
| on HN ]
Yes, that's another challenge with constructing benchmarks.
You could run the benchmarks using downstream providers like AWS if they are available.
You could vary the parameters of the tests to both help with LLMs learning as is and to help obscure that this is a test. Especially if sending them through a thirdparty, so the providers don't get identifying markers like IP addresses.
But ultimately it's another arms race like with viruses and anti-virus software.
Eh, I have an eval suite dating back to before GPT-5 in parts and have yet to see certain labs do as well on that as the public eval data would suggest.
It depends entirely on the financial incentive. Does improving against your eval suite lead to more adoption? Or would focusing on the far more popular benchmarks convince more CTOs to sign a multi-year contract?
Who'd sign a multi-year contract based on any benchmarks, public, private, mine or someone elses? We get new frontier LLMs every 0.5-2 months and any CTO of a major entity already uses Bedrock or the like, so why would anyone lock themselves into a provider or even worse a single model for multiple years?
Because API usage is extremely expensive and subscriptions are subsidized? We (my company of 100+) just moved off of AWS Bedrock to a locked in plan. Even when I was working for a much larger company (10k+ employees), we had enterprise subscriptions to OpenAI and Anthropic because the token cost for APIs was enormous.