News for hackers. Stuff that matters.
Re:Qwen4-Max by irthomasthomas on Monday November 16, 2026 @05:37AM (#50516108)
The private benchmarks can be seen by the labs when you run them. We only have their word that they wont look at them, which isn't worth much. We should assume that everyone benchmaxxes equally, apart from a few outliers, and try to climb the hill by producing relevant training data.
Re:Qwen4-Max by rhdunn on Monday November 16, 2026 @05:51AM (#50516182)
Yes, that's another challenge with constructing benchmarks.

You could run the benchmarks using downstream providers like AWS if they are available.

You could vary the parameters of the tests to both help with LLMs learning as is and to help obscure that this is a test. Especially if sending them through a thirdparty, so the providers don't get identifying markers like IP addresses.

But ultimately it's another arms race like with viruses and anti-virus software.

Re:Qwen4-Max by Topfi on Monday November 16, 2026 @08:33AM (#50517445)
Eh, I have an eval suite dating back to before GPT-5 in parts and have yet to see certain labs do as well on that as the public eval data would suggest.
Re:Qwen4-Max by paimapi on Monday November 16, 2026 @10:06AM (#50518384)
It depends entirely on the financial incentive. Does improving against your eval suite lead to more adoption? Or would focusing on the far more popular benchmarks convince more CTOs to sign a multi-year contract?
Re:Qwen4-Max by Topfi on Monday November 16, 2026 @10:29AM (#50518631)
Who'd sign a multi-year contract based on any benchmarks, public, private, mine or someone elses? We get new frontier LLMs every 0.5-2 months and any CTO of a major entity already uses Bedrock or the like, so why would anyone lock themselves into a provider or even worse a single model for multiple years?
Re:Qwen4-Max by paimapi on Monday November 16, 2026 @12:00PM (#50519831)
Because API usage is extremely expensive and subscriptions are subsidized? We (my company of 100+) just moved off of AWS Bedrock to a locked in plan. Even when I was working for a much larger company (10k+ employees), we had enterprise subscriptions to OpenAI and Anthropic because the token cost for APIs was enormous.