MaxxBench – "Nice DeepSWE score. But is it true?"

Re:MaxxBench – "Nice DeepSWE score. But is it true?" by druskacik on Saturday November 07, 2026 @04:50PM (#50408365)

Another hypothesis is that it's distilled models which collapse the most (small models from big labs are distilled versions of their large models).

It's not unreasonable to assume Kimi 3.1 is also a distilled model even though it's large (the difference being that it's a distilled from US models). I wouldn't be surprised if it also collapsed on this benchmark.

[ Parent | on HN ]

Re:MaxxBench – "Nice DeepSWE score. But is it true?" by Tade0 on Saturday November 07, 2026 @06:01PM (#50408879)

There's an interesting way to find this out: ask the model "What are you?" hundreds of times. Around 1% of replies at default temperature are really left field.

For instance, I had Qwen3.8 Flash Next reply with "I'm Kilo Code, a coding agent designed to ...".

That was in llama.cpp's UI - no harness whatsoever.

[ Parent | on HN ]

Re:MaxxBench – "Nice DeepSWE score. But is it true?" by Muromec on Saturday November 07, 2026 @08:48PM (#50409789)

DS4 when asked about censored stuff through api believed it's Claude. Otherwise it doesn't know what model it is (or pretends to not know).

[ Parent | on HN ]