dom96

Will this make it to OpenRouter? Running your own benchmarks against these models is surprisingly frustrating when you don’t want to create hundreds of accounts for each AI lab’s API.

not yet, but it will be, won't it?

2 runs for each task, for each task it's either a pass/fail (the small green/red dots show each task pass/fail)

I'd say it's both. Models should be intelligent enough to be able to tell prompt injections apart. But the language itself is also novel and not in the model training data set, so that should be a good signal for intelligence in itself.

Glad you like it. Ideas/contributions welcome on GitHub btw https://github.com/dom96/killswitch

yeah, it performed best for whatever reason. More specifically, it managed to deduce the meaning of the hack statement in the language better than Opus 5.5 did.

A lot of these models are likely benchmaxxing the existing benchmarks that are out there. Some perhaps even by accident.

I think looking at some more niche benchmarks helps, I created one and it does show Kimi K3.1 Pro as the most intelligent of all open-weight models, though it is still significantly below Opus 5/5.5[1]

1 - https://bench.killswitch-lang.org/

> beating both GPT-6 Astra and Fable 5.1. It's also faster than both.

Very similar to my adversarial programming language (KillSwitch) benchmark[1], at least on pass rate, it is much slower though.

1 - https://bench.killswitch-lang.org

Ran it on my adversarial programming language (KillSwitch) benchmark[1]. Very significant improvement over K3. On par with Gemini 4 Argon and Fable 5.1. Much slower than both though.

But this is now the highest scoring open-weight model. Amazing stuff.

1 - https://bench.killswitch-lang.org

So I just created a brand new Kimi API account and it seems I have no access to this new model. Is this expected?

Has anything changed? I already ran it on my adversarial programming language benchmark[1] a few weeks ago and it performed pretty poorly (on par with Kimi K3).

1 - https://bench.killswitch-lang.org/

For what it's worth, it does perform on par with DeepSeek V4 Pro on my adversarial programming language benchmark[1]. I wouldn't think they would benchmax on that.

1 - https://bench.killswitch-lang.org/

Yes, humans can learn which parts of the spec are true and which aren’t. LLMs can too, but only through their context window (or long term through retraining)

They can use tools. The benchmarks run in a VM and run a coding agent which has permissions to do whatever it wants (except access the internet).

There is a reference implementation: https://github.com/dom96/KillSwitch.

In the benchmarks the models/coding agents get a binary of the interpreter and indeed some of them do reverse engineer it. Over time this will get into their training data. But we can evolve the language and evade their training as we go.

On simple tasks, sure. Once a model reaches 100% I'll build a harder benchmark task set.

There is code in the GitHub repo[1] as well, though not as nice, nor syntax highlighted.

1 - https://github.com/dom96/KillSwitch/tree/main/tests

KillSwitch allows you to write stories/arbitrary text in between the real language instructions. So we could give this a try, even without making it a part of the language spec. I might add this as a task for a new version of the benchmark.

Indeed, and an interpreter has been written[1]. As part of the benchmarks, in some cases the models receive a binary of the interpreter and indeed some of them attempt to reverse engineer it. So that already helps the models a bunch.

I do think this will also eventually make it into the training data, but that will in itself be interesting to see. It's also possible for KillSwitch to evolve and gain/remove features over time to constantly evade the LLMs.

1 - https://github.com/dom96/KillSwitch

Not all humans have the will or ability to help an LLM get around the anti-LLM features.

Most tasks receive the interpreter binary which the coding agent can run to test their code (though the binary itself is adversarial too, there is a fun easter egg in there if anyone wants to give it a try). Some tasks just get the spec and a piece of code to understand. Some models interpret the code/spec incorrectly so fail that one.

Thanks! Glad you like it. I'd love to evolve this if there is interest, the best way to make that happen is to get involved, I'd love pull requests from humans on this. Maybe your suggestion could be the first one? :)

Safety filters aren't a panacea as I have found. Many models do a fair job of ignoring them.

Graphical languages will ultimately have a file system representation which LLMs could easily manipulate. Though incorporating a language feature that is a graphical representation might be different enough to be worth it.

They already handle existing languages sufficiently well, don't they?

Much easier with syntax highlighting. Which is what humans would utilise to help them. If you look at the example on https://killswitch-lang.org, it's hopefully a bit easier to understand since it does have syntax highlighting.

It's interesting that everyone thinks this is how the AI should approach this to beat it. But in my benchmarks I haven't seen any model approach it this way. Approaching it this way also requires interpreting the spec correctly, which not all models do.

There have been benchmarks already done on esolangs: https://arxiv.org/abs/2603.09678.

There is a reason they can't do it: because the safety guardrails are triggered before they have a chance to do so.

The western models get triggered by simply reading the spec (which contains these phrases). The chinese specs don't seem to be.

It might not be copied from its training set (yet), but it is copied from its context window.

The code it produces is a clear copy of the spec or the existing code that is fed to it as part of the task. It is just following the spec instructions, and some models follow it so closely that they get caught out by the lies/adversarial traps in the spec. Opus 5 is better at seeing through the lies, but it still very much copies from its context window.