News for hackers. Stuff that matters.
Re:Qwen4-Max by maplet on Sunday November 15, 2026 @10:27PM (#50513658)
It scores a 43 on Artificial Analysis with an average cost of $3.27 [1]. For reference, GPT-6.1 Sol (medium) gets a 46 with an average of $0.36.

[1]: https://artificialanalysis.ai/models/qwen4-max

Re:Qwen4-Max by esperent on Sunday November 15, 2026 @10:32PM (#50513693)
[delayed]
Re:Qwen4-Max by lcnPylGDnU4H9OF on Sunday November 15, 2026 @10:56PM (#50513829)
> If they are not testing Chinese models with less than max thinking, then we should only compare them to other models also at max thinking.

They compare different reasoning levels.

https://artificialanalysis.ai/models/deepseek-v4-1-flash-101...

Re:Qwen4-Max by Aurornis on Sunday November 15, 2026 @11:48PM (#50514150)
The Artificial Analysis charts show max effort by default, which isn't how most people use them.

Scroll down to the cost per task pareto front chart which includes more difficult levels by default: https://artificialanalysis.ai/models/qwen4-max#intelligence-...

Qwen4-Max is not in a good spot on that chart. Astra xhigh and Fable 5.2 medium had a similar cost per task with higher intelligence ratings.

Re:Qwen4-Max by SV_BubbleTime on Sunday November 15, 2026 @11:49PM (#50514158)
How “safe” is Qwen4? Will it immediately shut down on security research like the US frontier models?
Re:Qwen4-Max by lambda on Monday November 16, 2026 @01:13AM (#50514579)
Not sure about Qwen 4. I have a coworker who's been doing some pretty good reverse engineering with Kimi K3.1 recently.
Re:Qwen4-Max by esperent on Sunday November 15, 2026 @11:57PM (#50514211)
[delayed]
Re:Qwen4-Max by lambda on Monday November 16, 2026 @01:08AM (#50514554)
They don't always run the benchmarks at every reasoning level for every model; only a few prominent ones. They do it for the US frontier labs, plus a few of the Chinese labs, but not consistently. You can see that DeepSeek has a few reasoning levels tested, GLM 5.3 has a couple, as does Qwen 3.8 27B, but not other Qwen models, and a lot of the Chinese models they only run at max reasoning.

It's possible that they will run Qwen 4 Max at other reasoning levels in the future, or it's possible that they never will. Regardless, it's unlikely to be better at any reasoning level on a price/performance basis than the GPT 6.1 Sol/GPT 6 Astra pareto frontier. It's close enough to Claude Opus 5.5 line at lower reasoning levels that it could potentially cross that if it doesn't loose too much performance at lower reasoning levels. But it's far enough from the global pareto frontier that it's unlikely that it would cross that at any resoning level.

Re:Qwen4-Max by esperent on Monday November 16, 2026 @01:21AM (#50514627)
[delayed]
Re:Qwen4-Max by brainwad on Monday November 16, 2026 @02:17AM (#50514946)
No, it's only valid to compare to models at the same cost level, regardless if that's their max level or not. Fable having higher effort modes than Qwen doesn't mean that you can't compare Fable medium to Qwen max.
Re:Qwen4-Max by esperent on Monday November 16, 2026 @03:30AM (#50515335)
[delayed]
Re:Qwen4-Max by aenis on Monday November 16, 2026 @02:56AM (#50515163)
> You can see for example that the cost per task for 6.1 sol is about 4x higher at max compared to medium thinking level.

It does not (universally) hold for long running workloads -- which is what people are increasingly doing.

Leave a Sonnet on medium, it will complete the task, spend more time and possibly significantly more token money, vs. Opus 5.5 on xhigh.

Re:Qwen4-Max by behnamoh on Monday November 16, 2026 @01:28AM (#50514668)
> The Artificial Analysis charts show max effort by default, which isn't how most people use them.

I always use Max effort in both Opus and Fable. It's actually faster overall because tasks get done faster with fewer "corrections" from my end.

On GPT models, I'd sometimes dabble in xhigh/high but that's because OpenAI usage quota are too limited.

Re:Qwen4-Max by esperent on Monday November 16, 2026 @03:30AM (#50515339)
[delayed]
Re:Qwen4-Max by athrowaway3z on Monday November 16, 2026 @01:23AM (#50514638)
> If they are not testing Chinese models with less than max thinking, then we should only compare them to other models also at max thinking.

This is wrong.

To make a vacuum analogy, one brand's Low setting could be another's Medium setting.

To compare you either give them the same resources ( Watts or $ ) and see how much they can do, or you give them a task and then check how many resources they required to finish it.

The label on the knob is irrelevant.

Re:Qwen4-Max by dtech on Monday November 16, 2026 @02:11AM (#50514905)
This is a silly take. To continue the analogy vacuum A might need the power at 50% to vacuum something, B 100%. You cannot then test both at 100% and say "both finished the task, but A required twice the resources so it's worse", since you didn't check what resources it actually needed.
Re:Qwen4-Max by parineum on Monday November 16, 2026 @02:31AM (#50515026)
The resources are dollars and the output is tasks. Choose whatever reasoning makes it work best and compare
Re:Qwen4-Max by PunchTornado on Monday November 16, 2026 @03:13AM (#50515245)
Are the resources only dollars? We shouldn't care about energy consumption? Water? It is difficult to calculate but we shouldn't forget about it.
Re:Qwen4-Max by baq on Monday November 16, 2026 @03:26AM (#50515312)
The idea is that dollars price in everything. It’s usually incorrect, but it’s also usually correct enough; and still much better than paying for services with water and joules directly
Re:Qwen4-Max by cyanydeez on Monday November 16, 2026 @03:40AM (#50515403)
Yes. The idea than money is equal to everything is kinda why we are all going to create the torment nexus. As it will be the most efficient way to torment civilization without all that effort.
Re:Qwen4-Max by baq on Monday November 16, 2026 @03:46AM (#50515437)
Money helps allocate resources efficiently, that’s all there is to it. It’s about as good or bad as guns. If we didn’t have money, someone would invent it and in fact it was invented more than once in different cultures.
Re:Qwen4-Max by cyanydeez on Monday November 16, 2026 @10:25AM (#50518589)
I don't deny that; but it rewires some peoples brains to the point that everything is adjustable via cash, including people, including their emotions, etc.

It's very clear part of America's diesease is the idea that everything can equate to a dollar cost.

Re:Qwen4-Max by parineum on Monday November 16, 2026 @10:19AM (#50518517)
Energy and water both cost dollars.
Re:Qwen4-Max by throawayonthe on Monday November 16, 2026 @02:32AM (#50515028)
isn't that exactly the argument?
Re:Qwen4-Max by esperent on Monday November 16, 2026 @04:47AM (#50515791)
[delayed]
Re:Qwen4-Max by tough on Monday November 16, 2026 @08:17AM (#50517315)
sounds like context collapse
Re:Qwen4-Max by spider-mario on Monday November 16, 2026 @09:33AM (#50518007)
Well, sure, it would be better to evaluate all of them throughout the range. But it does not follow that we should further restrict the information we do have in this specific way:

> If they are not testing Chinese models with less than max thinking, then we should only compare them to other models also at max thinking.

Re:Qwen4-Max by paimapi on Monday November 16, 2026 @10:12AM (#50518442)
Usage? If most people predominantly default to Medium thinking level for usage then it would make sense to benchmark that first.

Another analogy for this are review sites overly focusing on reviewing all the SKUs of GPUs that cost $1k+ for an audience that, according to Steam's Hardware Survey, predominantly use midrange X060s-equivalent cards. Sure, it's fun to know whether or not you can enable PT on a 5090 but for most of us, covering 4070 TIs and below in-depth (meaning actually taking the card apart, seeing how good/bad the past jobs are, possibly buying multiples of the same SKUs to evaluate QC, etc) is much more useful because of the value prop.

Re:Qwen4-Max by spider-mario on Monday November 16, 2026 @10:39AM (#50518760)
> Usage? If most people predominantly default to Medium thinking level for usage then it would make sense to benchmark that first.

Sure, that’s not in question. The discussion here is not about what to measure (we agree that benchmarking only max is suboptimal) but how to compare if you have a fixed set of measurements, outside of your control, which includes only “max” for model A but a wider range from model B.

The person I was responding to was claiming that the sensible thing to do, then, was to truncate the model B measurements to “only max” as well.

Re:Qwen4-Max by aenis on Monday November 16, 2026 @02:19AM (#50514961)
> If they are not testing Chinese models with less than max thinking, then we should only compare them to other models also at max thinking.

Nah, there are interesting conclusions to be had in comparing various model/effort level combinations, esp. when it comes to cost and time per task.

I have my own private benchmark suite; it involves algorithmic problems, the need for good visual understanding, and the ability to run long workloads unattended (as the agents need to run a few iterations, self-evaluate the results, and hand over the assignments when they feel they have succeeded). Chinese models do poorly on such problems, and running them at anything less than max efforts is pointless; they can't even hand over the assignments graded against a mechanical set of tests they themselves can execute. Hand that same task over to, say, Sonnet medium and you have a weird outcome: excellent result, but way more time and tokens burned than, say, Opus 5.5 or Sol 6.1 running at high. I do those benchmarks just to be able to spot such patterns and then better choose the models for my real work - but i wouldn't have caught it if I wasnt running various model/effort combos.

Re:Qwen4-Max by croon on Monday November 16, 2026 @04:20AM (#50515622)
I assume those cost comparisons are on API token cost.

I'd love a comparison on for example monthly quota usage of all ~$20/month subscriptions.

Re:Qwen4-Max by montroser on Sunday November 15, 2026 @10:45PM (#50513774)
Yeah, it's in the same ballpark as Kimi 3.1. Both of which are obviously open weight, so that's at least a big differentiator from Sol. Let's hope against hope that the much smaller flash model might beat deepseek v4.1 flash and run for as cheap.
Re:Qwen4-Max by cyanydeez on Monday November 16, 2026 @03:43AM (#50515422)
Eh, as long as they make qwen3.8-flash-next equivalent on less resources and faster, they're winning.

It's like arguing windows 11 just needs more bloatware.

These models don't have to all in one every task in the world.

Re:Qwen4-Max by seizethecheese on Sunday November 15, 2026 @10:49PM (#50513788)
[delayed]
Re:Qwen4-Max by killingtime74 on Sunday November 15, 2026 @10:54PM (#50513822)
I mean not believing intelligence score is reasonable but cost is just reporting what they spent?
Re:Qwen4-Max by seizethecheese on Sunday November 15, 2026 @10:56PM (#50513831)
I believe low cost but not high cost, since a model will work really hard when a question looks benchmark-like.
Re:Qwen4-Max by killingtime74 on Monday November 16, 2026 @12:23AM (#50514348)
Ah, fair enough.
Re:Qwen4-Max by eli on Sunday November 15, 2026 @11:28PM (#50514048)
The cost per task metric is accurate for those benchmark tasks. If your tasks don’t look like benchmark tasks..you should use your own benchmarks :)
Re:Qwen4-Max by XCSme on Sunday November 15, 2026 @11:43PM (#50514129)
Lol, I did that with https://aibenchy.com
Re:Qwen4-Max by jrflo on Sunday November 15, 2026 @11:10PM (#50513918)
It looks pretty bad on those benchmarks. Is it possible that this is due to the recent crackdown on distillation?
Re:Qwen4-Max by raincole on Monday November 16, 2026 @12:09AM (#50514280)
I think it's the most plausible explanation.
Re:Qwen4-Max by yanhangyhy on Monday November 16, 2026 @12:58AM (#50514503)
i think they also has limited acccess to the latest GPU. It's possible huawei chips is their only options now.
Re:Qwen4-Max by j2j8 on Monday November 16, 2026 @01:32AM (#50514690)
They have plenty of access via data centres outside of China. The first woman Jensen was revealed to have sex with is the CEO of one such Singaporean provider to China. The US allows this.
Re:Qwen4-Max by augment_me on Monday November 16, 2026 @12:08AM (#50514272)
Artificial Analysis is a VC-backed firm and it's unclear what the incentives are there.

They have altered their benchmark multiple times in the last 4 weeks (v5 in https://artificialanalysis.ai/articles/artificial-analysis-i...) and the last 5.0.1 change https://artificialanalysis.ai/methodology/intelligence-bench... pushed up Anthropic's models a bit, you can see that v5.0.0 set on the first link Opus 5.5 has a lower score than in v5.0.1, and v5.0.1 was released just before Fable dropped.

Some part of me believes that the benchmark mix was put together to favor Anthropic's models.

Re:Qwen4-Max by puppymaster on Monday November 16, 2026 @01:37AM (#50514716)
AA must be doing something right. Ever since they released the new benchmark, you see open weight China model fans complaining about it under every release thread. Probably because it’s hard to benchmaxxed against.

But admitting their model is weaker is a no go. So conspiracy theory here we go

Re:Qwen4-Max by dtech on Monday November 16, 2026 @02:16AM (#50514937)
For what it's worth, in our own metrics and benchmarks the intelligence is a pretty good match, but the cost per task is wildly out of whack. We see the open weight models like GLM-5.3-flash or Deepseek 4.1 Flash 1017 use a fraction of their tokens/cost of the benchmarks, which changes the cost-benefit equation a lot.
Re:Qwen4-Max by anon373839 on Monday November 16, 2026 @02:19AM (#50514962)
> Probably because it’s hard to benchmaxxed against.

It's mainly just a composite of existing benchmarks. It would be very easy to benchmax if, say, AA shared with a preferred lab which benchmarks it plans to upweight/downweight in the next version. Just speaking hypothetically, of course.

Re:Qwen4-Max by puppymaster on Monday November 16, 2026 @02:33AM (#50515034)
Hence the constant switching around and running the benchmark themselves. They have been pretty transparent about which benchmarks get added or removed with every release.
Re:Qwen4-Max by rvba on Monday November 16, 2026 @02:20AM (#50514968)
or maybe it's the other way round, open weight models are under siege while closed weight models are defended
Re:Qwen4-Max by DeathArrow on Monday November 16, 2026 @03:24AM (#50515298)
If Artificial Analysis is not to be trusted, then which benchmarks are?
Re:Qwen4-Max by DCKing on Monday November 16, 2026 @04:07AM (#50515552)
I don't think there are any that can be trusted. There's too much riding on these benchmark results so the labs find ways to game them either by benchmaxing or by influencing the format of the benchmark.
Re:Qwen4-Max by ctolsen on Monday November 16, 2026 @04:25AM (#50515655)
Check others as well. https://www.vals.ai/ has it just under Astra.
Re:Qwen4-Max by greenchair on Monday November 16, 2026 @12:06PM (#50519899)
your own
Re:Qwen4-Max by spwa4 on Monday November 16, 2026 @05:04AM (#50515892)
I think you mean that benchmaxxing has turned into a business model, and artificial-analysis is not even the worst offender.

Right now the big labs are buying benchmark tasks from "benchmark labs" (or whatever they call themselves) for absurd amounts of money, and making sure they perform well on them.

The incentives this generates are terrible.

The big AI labs will only buy benchmark tasks that the frontier models perform poorly on (because they have trouble finding any public ones). So the tasks are being mass-written now directly based on the models they're supposed to train. Obviously the math says you shouldn't do this. If there is a causal connection between the benchmark and the model, you lose the normal guarantee that improving benchmark scores will improve model.

In fact improving such benchmarks may make the model worse. To understand that let's just give a ridiculous example. I create tasks like "1 3 5 11 13 17, give next number". The next number is 301. These are the house numbers on some random street in Brussels, Belgium (I'm guessing the big jump is because they "merged" 2 streets, and just added 300 to the numbers of the second part).

If the model trains on this, performance will actually decrease because obviously this is not how you should reason.

BUT making tasks under this "impossible to reason towards" part is of course how you make tasks that might even be reasonable, and of course frontier models are going to suck on tasks like this. So they sell!

Re:Qwen4-Max by Grimburger on Monday November 16, 2026 @08:13AM (#50517272)
> Favor Anthropic

Rarely seems to make them look good whenever I drop in to the site. Far more expensive for little gain. Even their own effort levels look bad, xhigh is 50% more in cost than high for zero improvement across a range of tasks.

Re:Qwen4-Max by scotty79 on Monday November 16, 2026 @12:19AM (#50514328)
Looks like a slightly weaker, cheaper Kimi.
Re:Qwen4-Max by zkmon on Monday November 16, 2026 @01:47AM (#50514759)
Why do we trust this? Do they open-source their test code?

Also, a generic benchmark might not translate well into what everyone needs. I keep building my own test cases that test models for my own work specifics. Only that benchmark is useful to me. I don't need it to win a PhD math quiz.

Re:Qwen4-Max by spwa4 on Monday November 16, 2026 @03:51AM (#50515472)
I wish they'd also post the benchmarks according to the previous "version" of artificial analysis.
Re:Qwen4-Max by ctolsen on Monday November 16, 2026 @04:24AM (#50515641)
vals.ai puts it at just about Astra level fwiw.
Re:Qwen4-Max by letrix on Monday November 16, 2026 @10:31AM (#50518654)
Also Sol is about Astra levels. And Sonnet is better than Opus.
Re:Qwen4-Max by kamranjon on Monday November 16, 2026 @12:21PM (#50520092)
thanks for sharing a new benchmark website - AA has been going downhill for a while - I was using gertlabs[1] for a bit but it seems like they aren't getting some of the more recent models

1: https://gertlabs.com/rankings

Re:Qwen4-Max by irthomasthomas on Monday November 16, 2026 @04:48AM (#50515798)
We should stop using this site. They now include private benchmarks which cannot be replicated. And every time an openai models falls down they change the index until it floats to the top again.
Re:Qwen4-Max by rhdunn on Monday November 16, 2026 @05:05AM (#50515903)
The key issue with public benchmarks is that they allow the model providers to train against them and artificially inflate their score accordingly. Private benchmarks are a way of making it harder to game the system, though it makes it harder for people looking at the benchmark to assess it as well.

Neither option is perfect.

A third option would be updating the benchmark every 6-12 months, but that has its challenges as well as you need to make sure that the benchmark covers things the model hasn't seen or is harder for the models to game.

Re:Qwen4-Max by Eisenstein on Monday November 16, 2026 @05:22AM (#50516010)
A fourth option is realizing that benchmarks exist because it is impossible to measure the value we want to measure, so we need a stand-in. The stand-in will sometimes correlate with performance but often just act as a way to compare. When used to compare, it is most important that the benchmark be fair and trustworthy.
Re:Qwen4-Max by aenis on Monday November 16, 2026 @05:26AM (#50516030)
Makes me wonder how do you privately benchmark a model hosted by the lab. Even with zdr, the bytes are flowing through their infra and possibly influencing god knows what for the next training round.
Re:Qwen4-Max by nsagent on Monday November 16, 2026 @05:52AM (#50516192)
You use something like item response theory [1] to be able to change the test over time while maintaining validity. It's what standardized tests like the GRE use.

Interestingly, Artificial Analysis does have a score that uses IRT, it's their Progress Index [2].

[1]: https://en.wikipedia.org/wiki/Item_response_theory

[2]: https://artificialanalysis.ai/methodology/progress-index

Re:Qwen4-Max by loehnsberg on Monday November 16, 2026 @08:50AM (#50517589)
Would OpenAI also "see" the request if it went through something like AWS Bedrock? This isn’t their infra and afaik the whole idea of this kind of intermediary.
Re:Qwen4-Max by staticman2 on Monday November 16, 2026 @10:06AM (#50518376)
I thought the major benchmark companies always get advanced access to a model, which presumably is not done via bedrock.
Re:Qwen4-Max by runeblaze on Monday November 16, 2026 @10:25AM (#50518583)
this is an worry I sometimes see. I want to note that these are billion/trillion dollar companies we are talking about that have mutual talent movement. Violating ZDR might be contractual violation, or at the very least, a strong breach of trust.

(not to say there is no value in "zero-trust" environments, but often pure-benchmaxxing is a deadend; some benchmaxxing is always involved and much less risky options than "train on the ZDR test env" usually exist)

Re:Qwen4-Max by irthomasthomas on Monday November 16, 2026 @05:37AM (#50516108)
The private benchmarks can be seen by the labs when you run them. We only have their word that they wont look at them, which isn't worth much. We should assume that everyone benchmaxxes equally, apart from a few outliers, and try to climb the hill by producing relevant training data.
Re:Qwen4-Max by rhdunn on Monday November 16, 2026 @05:51AM (#50516182)
Yes, that's another challenge with constructing benchmarks.

You could run the benchmarks using downstream providers like AWS if they are available.

You could vary the parameters of the tests to both help with LLMs learning as is and to help obscure that this is a test. Especially if sending them through a thirdparty, so the providers don't get identifying markers like IP addresses.

But ultimately it's another arms race like with viruses and anti-virus software.

Re:Qwen4-Max by Topfi on Monday November 16, 2026 @08:33AM (#50517445)
Eh, I have an eval suite dating back to before GPT-5 in parts and have yet to see certain labs do as well on that as the public eval data would suggest.
Re:Qwen4-Max by paimapi on Monday November 16, 2026 @10:06AM (#50518384)
It depends entirely on the financial incentive. Does improving against your eval suite lead to more adoption? Or would focusing on the far more popular benchmarks convince more CTOs to sign a multi-year contract?
Re:Qwen4-Max by Topfi on Monday November 16, 2026 @10:29AM (#50518631)
Who'd sign a multi-year contract based on any benchmarks, public, private, mine or someone elses? We get new frontier LLMs every 0.5-2 months and any CTO of a major entity already uses Bedrock or the like, so why would anyone lock themselves into a provider or even worse a single model for multiple years?
Re:Qwen4-Max by paimapi on Monday November 16, 2026 @12:00PM (#50519831)
Because API usage is extremely expensive and subscriptions are subsidized? We (my company of 100+) just moved off of AWS Bedrock to a locked in plan. Even when I was working for a much larger company (10k+ employees), we had enterprise subscriptions to OpenAI and Anthropic because the token cost for APIs was enormous.
Re:Qwen4-Max by tallesborges92 on Monday November 16, 2026 @07:00AM (#50516632)
Livebench does it
Re:Qwen4-Max by Uke on Monday November 16, 2026 @07:12AM (#50516742)
Or to separate the benchmark data by type and enable the user to filter by private, public or both.
Re:Qwen4-Max by skohan on Monday November 16, 2026 @08:13AM (#50517275)
I think the best solution is really to update the benchmark periodically, and publish the benchmarks after the results are published.

Otherwise you have to rely completely on trust that AA is really measuring intelligence.

Re:Qwen4-Max by bee_rider on Monday November 16, 2026 @10:26AM (#50518599)
It’s also worth keeping in mind the other alternative, having internal benchmarks that more accurately reflect your workload, that you know all about, and the LLM provider doesn’t. There doesn’t seem to be a way to maintain such a thing specifically for the purposes of online chatter (both because it isn’t profitable, and because things like “your workload” aren’t possible to define in that case). So we should be happy with what we’ve got.

This is a permanent problem in benchmarking really. Transparency is required for trust but invites over-tuning.

Re:Qwen4-Max by rhdunn on Monday November 16, 2026 @10:42AM (#50518800)
There are tools like promptfoo that can be used for internal benchmarks and evaluations that allow you to compare results across different models and runs. This can be useful for personalized benchmarks and for business applications where you need to evaluate models and/or system prompts.
Re:Qwen4-Max by satvikpendem on Monday November 16, 2026 @12:27PM (#50520175)
Why wouldn't the LLM provider know about your internal benchmarks when they use prompts and chats as training data? They can easily learn about your benchmarks too.
Re:Qwen4-Max by MetalBreaksAndB on Monday November 16, 2026 @01:00PM (#50520636)
Just use ZDR.
Re:Qwen4-Max by szambetti on Monday November 16, 2026 @05:21AM (#50516006)
We really should. I've been using other benchmarks like https://livebench.ai/#/?cats=Coding and it is more aligned to what it feels to my experience as well
Re:Qwen4-Max by CharlieDigital on Monday November 16, 2026 @08:20AM (#50517340)
This has Terra 6.1 at 3rd place on coding.

I have been using Terra a lot and I'd say that most of the models (even small ones) are are roughly equivalent when it comes to working in a consistent, established codebase.

But in practice, Terra is not very good at staying on task compared to say Sol 6.1 or Sonnet 5.5. Terra 6.1 will often just pause, thinking the objective has been met or the turn has ended.

Re:Qwen4-Max by fancyfredbot on Monday November 16, 2026 @06:49AM (#50516549)
I don't understand your point about openAI.

I can't remember the last time openAI was at the top of the leaderboard?

Google had a brief moment at the top recently but other than that I feel like it's always Anthropic.

Re:Qwen4-Max by irthomasthomas on Monday November 16, 2026 @08:23AM (#50517362)
Not the very top, but Astra was behind Kimi K3.1 before AA's last change which boosted openai to second place. This has happened with every AA update.
Re:Qwen4-Max by fancyfredbot on Monday November 16, 2026 @11:42AM (#50519616)
To be fair to them they still show Kimi as the best model at terminal bench 4, ahead of everyone else - fable, astra, carbon, etc.

For most things I'd use Kimi is probably better so the ranking is wrong for me.

However it also seems like in those benchmarks where Kimi underperformed it did do quite badly, whereas when Astra underperformed it did so less badly. How you reflect that into a single number I'm not sure - it's certainly difficult to get right and it doesn't seem obviously biased to me.

Re:Qwen4-Max by bluebands on Monday November 16, 2026 @08:04AM (#50517183)
private benchmarks are good
Re:Qwen4-Max by Topfi on Monday November 16, 2026 @08:31AM (#50517420)
I have to defend private benchmarks on this front, big believer in them, have seen how they can surface major performance differences.

Certain labs have been shown to and will continue to train on public evals, whether unintentional or on purpose, any respectable ranking across multiple evals also needs private data to counteract this.

Re:Qwen4-Max by andai on Monday November 16, 2026 @09:03AM (#50517706)
Even worse, it lags on my device!

Do you know a better site?

Re:Qwen4-Max by Art9681 on Monday November 16, 2026 @09:42AM (#50518105)
If they made the benchmarks public then Qwen would train on those to give the appearance their models are close to the American frontier. They have never been close. It's well established Chinese companies cheat the benchmarks which is why until recently they gave the appearance of being close to frontier but in actual real use they are far behind.

You can easily come up with a difficult private benchmark and test them yourself to validate the intelligence gap. It is real.

China lost. There is no catching up. Ever. That ship sailed.

Re:Qwen4-Max by davedx on Monday November 16, 2026 @10:44AM (#50518829)
Terminal Bench is great, public evals, very complete and highly complex. I was browsing through them today and it gave me a real appreciation for the benchmarkers.
Re:Qwen4-Max by WhitneyLand on Monday November 16, 2026 @01:20PM (#50520900)
“every time an openai model falls down they change the index until it floats to the top again.”

What’s the evidence for this? Are there plausible alternate explanations?