There was also a bug in which the reasoning tokens were “encrypted” for opus but you could ask Haiku to regurgitate them and it would.
Anthropic have been doing anything and everything to prevent distillation “usage” and it’s caused me at least once to cancel my subscription because it’s hard to trust a company that secretly alters its responses based on something I can’t see and never get feedback on.
Asking it to think/reason too strongly would hit classifiers. Quoting reasoning summaries in my messages would hit classifiers. It thinking about how post-training effects another model's would hit classifiers.
I thought it might be overzealous Fable 5.2 protection, but at one point I went through a chain of Fable -> Opus -> Sonnet. Eventually even Sonnet triggered a classifier no matter how the conversation went... I had to rewind and have Claude prompt Codex going forward for most of the work.
Yet every time I compare the two in something like Pi, Sol ends up being 5-10x the price for a simple query, something like 7-10c vs. less than 0.5c for DeepSeek. What's the deal?
Models often get great scores despite being a lot worse than the frontier in practice.
AA has always been miningless, but people keep referring yo it anyway just because the synthetic view is too convenient (even if meaningless).
they keep harnesses the same across models other than in the coding agent index (https://artificialanalysis.ai/agents/coding-agents), which uses native harnesses for all models too
Ah yes, the reverse-benchmaxers of OpenAI models.
[1] Especially Terminal-Bench-Science 0.1 stands out. Example problems:
- https://hub.harborframework.com/tasks/terminal-bench-science...
- https://hub.harborframework.com/tasks/terminal-bench-science...
Probably very relevant to people working in these fields, but it is honestly not surprising if a lab with limited compute will not train for these technologies, and it says pretty much nothing about more common use cases. This is where you really need a breakdown of capabilities and not an increasingly meaningless aggregate. I also suspect that this benchmark measures knowledge retrieval more than raw reasoning ability.
CCP can probably "volunteer" a couple of thousand to build a high quality national training data bank.
Maybe somebody should archive the scores and see if they in the future ever backtrack particular models to favour others (like last week)