Re:Qwen4-Max
by Topfi on Monday November 16, 2026 @08:33AM
(#50517445)
Eh, I have an eval suite dating back to before GPT-5 in parts and have yet to see certain labs do as well on that as the public eval data would suggest.
[ Parent
| on HN ]
It depends entirely on the financial incentive. Does improving against your eval suite lead to more adoption? Or would focusing on the far more popular benchmarks convince more CTOs to sign a multi-year contract?
Who'd sign a multi-year contract based on any benchmarks, public, private, mine or someone elses? We get new frontier LLMs every 0.5-2 months and any CTO of a major entity already uses Bedrock or the like, so why would anyone lock themselves into a provider or even worse a single model for multiple years?
Because API usage is extremely expensive and subscriptions are subsidized? We (my company of 100+) just moved off of AWS Bedrock to a locked in plan. Even when I was working for a much larger company (10k+ employees), we had enterprise subscriptions to OpenAI and Anthropic because the token cost for APIs was enormous.