News for hackers. Stuff that matters.
Re:Qwen4-Max by anana_ on Sunday November 15, 2026 @10:03PM (#50513527)
Artificial Analysis score doesn't look too hot. Doesn't even surpass Kimi 3.1
Re:Qwen4-Max by selectodude on Sunday November 15, 2026 @10:05PM (#50513539)
Anthropic finally figured out how to block them from distilling Opus.
Re:Qwen4-Max by m00dy on Sunday November 15, 2026 @10:24PM (#50513645)
how so ?
Re:Qwen4-Max by gardnr on Monday November 16, 2026 @12:37AM (#50514407)
Anthropic has been trying many different ways to prevent this, including a proposal to nerf your sessions if they “detected” malicious activities (anyone’s coding sessions would “degrade” if detected as malicious). This led to a huge outcry from developers complaining that they “need to be able to trust” the responses from Claude and intentionally poisoning responses in some sort of weird shadow ban was unacceptable. They walked it back but the arc leak also showed some dubious weird regexes to nerf “malicious” users. They have a real distillation problem. It’s hard to have empathy for a billion dollar company that distilled the contents of the internet into a thing and now is crying foul when someone distills their outputs.

There was also a bug in which the reasoning tokens were “encrypted” for opus but you could ask Haiku to regurgitate them and it would.

Anthropic have been doing anything and everything to prevent distillation “usage” and it’s caused me at least once to cancel my subscription because it’s hard to trust a company that secretly alters its responses based on something I can’t see and never get feedback on.

Re:Qwen4-Max by m00dy on Monday November 16, 2026 @01:03AM (#50514523)
what do you think the current status of this like encrypting reasoning tokens along the way ?
Re:Qwen4-Max by BoorishBears on Monday November 16, 2026 @04:52AM (#50515824)
This weekend I had the audacity to use Claude Code to help with the code side of post-training a model...

Asking it to think/reason too strongly would hit classifiers. Quoting reasoning summaries in my messages would hit classifiers. It thinking about how post-training effects another model's would hit classifiers.

I thought it might be overzealous Fable 5.2 protection, but at one point I went through a chain of Fable -> Opus -> Sonnet. Eventually even Sonnet triggered a classifier no matter how the conversation went... I had to rewind and have Claude prompt Codex going forward for most of the work.

Re:Qwen4-Max by omnimus on Monday November 16, 2026 @03:00AM (#50515184)
Of course all the work has always been done by US companies. China labs are just copycats.
Re:Qwen4-Max by anon373839 on Sunday November 15, 2026 @10:07PM (#50513556)
AA has become pretty meaningless. They just keep adjusting the recipe to get a preferred ranking result, with no supporting evidence for their decisions.
Re:Qwen4-Max by gr_norm on Sunday November 15, 2026 @10:30PM (#50513677)
Yeah, I don't know why people bother with it anymore. It no longer reflects what you're likely to see in real-world usage, at least IME.
Re:Qwen4-Max by stratts on Sunday November 15, 2026 @10:31PM (#50513686)
One thing I find confusing about AA is that they show GPT 6.1 dominating the lower end of the Pareto line, with eg, GPT 6.1 Sol at medium supposedly costing far less per task than DeepSeek V4.1 Flash at high.

Yet every time I compare the two in something like Pi, Sol ends up being 5-10x the price for a simple query, something like 7-10c vs. less than 0.5c for DeepSeek. What's the deal?

Re:Qwen4-Max by blackoil on Sunday November 15, 2026 @10:37PM (#50513725)
Same for me coding tasks taking $20-50 are easily done by deepseek in $5-10
Re:Qwen4-Max by the_duke on Sunday November 15, 2026 @10:43PM (#50513760)
To be fair, a lot of the popular benchmarks have become pretty meaningless.

Models often get great scores despite being a lot worse than the frontier in practice.

Re:Qwen4-Max by anon373839 on Monday November 16, 2026 @12:09AM (#50514288)
This is true, but if you look at the change in weighting to specific benchmarks from one version of AA to the next, it looks really arbitrary.
Re:Qwen4-Max by stymaar on Monday November 16, 2026 @02:35AM (#50515043)
A benchmark is only relevant if it was released after the model it evaluates.
Re:Qwen4-Max by wasabi991011 on Monday November 16, 2026 @12:49AM (#50514467)
I thought they were an independent company, what motive would they have to prefer some ranking results over others?
Re:Qwen4-Max by stymaar on Monday November 16, 2026 @02:33AM (#50515038)
> AA has become pretty meaningless

AA has always been miningless, but people keep referring yo it anyway just because the synthetic view is too convenient (even if meaningless).

Re:Qwen4-Max by mysteriousbow on Monday November 16, 2026 @11:19AM (#50519294)
> test Anthropic and OpenAI in their native harnesses, while not doing the same for Qwen/Deepseek/etc.

they keep harnesses the same across models other than in the coding agent index (https://artificialanalysis.ai/agents/coding-agents), which uses native harnesses for all models too

Re:Qwen4-Max by AAisthetiobeofA on Sunday November 15, 2026 @10:26PM (#50513652)
> Artificial Analysis

Ah yes, the reverse-benchmaxers of OpenAI models.

Re:Qwen4-Max by yipinwong on Sunday November 15, 2026 @11:05PM (#50513885)
Ah yes, another new account without any backup to the claim.
Re:Qwen4-Max by rbehrends on Sunday November 15, 2026 @10:45PM (#50513770)
While I doubt that Qwen 4 Max can seriously compete with frontier models, the AA index has become all but useless, due to the introduction of new benchmarks in the aggregate which are of questionable value as a measure of "intelligence" [1]. AA has seemingly also abolished updating the Coding Agents Index, which was an okay benchmark for coding ability, and older models are completely absent from the v5 version of the full index.

[1] Especially Terminal-Bench-Science 0.1 stands out. Example problems:

- https://hub.harborframework.com/tasks/terminal-bench-science...

- https://hub.harborframework.com/tasks/terminal-bench-science...

Probably very relevant to people working in these fields, but it is honestly not surprising if a lab with limited compute will not train for these technologies, and it says pretty much nothing about more common use cases. This is where you really need a breakdown of capabilities and not an increasingly meaningless aggregate. I also suspect that this benchmark measures knowledge retrieval more than raw reasoning ability.

Re:Qwen4-Max by izabera on Sunday November 15, 2026 @11:19PM (#50513980)
alibaba has "limited compute"?
Re:Qwen4-Max by rbehrends on Sunday November 15, 2026 @11:26PM (#50514034)
Compared to OpenAI and Anthropic, yes. There's this export ban on Nvidia chips, after all. And all the CPUs they have won't help much with model training.
Re:Qwen4-Max by mitthrowaway2 on Monday November 16, 2026 @01:26AM (#50514660)
Can't they rent GPU time in American data centers like everyone else does?
Re:Qwen4-Max by j2j8 on Monday November 16, 2026 @01:38AM (#50514720)
They do in huge Singaporean ones like Megaspeed.
Re:Qwen4-Max by mrheosuper on Monday November 16, 2026 @04:53AM (#50515827)
That and limited training data.
Re:Qwen4-Max by throwa356262 on Monday November 16, 2026 @08:15AM (#50517300)
I am surprised that this is a problem for China.

CCP can probably "volunteer" a couple of thousand to build a high quality national training data bank.

Re:Qwen4-Max by gardnr on Monday November 16, 2026 @12:27AM (#50514356)
Did AA take on funding or something?
Re:Qwen4-Max by alfiedotwtf on Monday November 16, 2026 @12:50AM (#50514473)
Not only that. I find it weird that some models like Laguna S 2.1 were never included.

Maybe somebody should archive the scores and see if they in the future ever backtrack particular models to favour others (like last week)