Of course this is authored by people at Anthropic. Continuing their narrative that open sourced models are dangerous and that people should trust the closed sourced providers.

This of course is not an issue if you actually care about security where you would've already sandboxed your agent and put guardrails on the inputs/outputs of the model.

> Poisoning only 0.1% of the pretraining corpus

Considering the scale of the datasets used by most companies, being able to poison one out of every thousand documents used in it is a huge leap.

Then, on top of that, you have to also then get your trigger somehow sent to an agent that will honor it.

I guess I'm just not impressed? Like yeah, if you train a model to do bad things it will do bad things, that shouldn't be shocking to people.

I think open source models could be vulnerable to these attacks. So long as I sandbox everything then I am as secure as my sandbox which is air gapped from my personal stuff i.e. no models on the same machine I log into my bank.

I know closed source models will have nefarious shit injected into it. It will start modest at first, of course, they need to get past the hyper scrutiny phase. Then it will simply be telemetry, then it will simply be feeding my behaviors into ad targeting, then it will simply be computer vampirism, and so on. One step at a time.

I prefer a gremlin I know I need to keep chained than a seductress I think I can probably trust.

How is this specific to open source models?

Uno reverse it: “… may be installing backdoors in people’s code for the government and may have just published a blueprint on how this is achieved.”

There’s nothing open model specific about this paper. They do leave it pretty open to people who are concerned about the Chinese government, but then again you don’t see anyone saying that people who advise people visiting China to use burner phones are Anthropic shills.

Or, for that matter, accusing people who recommend burner phones for visiting the US of shilling for China. Sometimes geopolitical caution is warranted.

[delayed]

I've been thinking about this for about a year. In my head I've called it "context bombs", though that may not be super accurate.

Think back to the movie The Manchurian Candidate. Produce highly capable LLM models then give them away for free. Train them with key phrases/codes etc to perform nefarious actions later when those codes are present in some piece of ingested context.

There were a couple of papers written on the topic in 2018 (I think, I'll need to find them again) and there's been surprisingly little more work done on the topic since. At least from what I've seen.

I really love the use of local models over subscription services but personally I would never recommend using open weights models from certain sources in a commercial environment.

I just sort of presumed this was already being done. Because why wouldn't it be?

Well, if it is being done, per the other article on the front page, we should be able to find it.

If you were a malicious actor do you really think this method is going to offer the greatest return on investment?

Nation states aren’t looking for a return on investment rather they have a particular objective they want to achieve, e.g. stuxnet or the pager attack.

Sew a million seeds, see what grows.

I think this is a reasonable question. The moment this activates and exfiltrates or destroys a system the entire model family, training pipeline, and company are burned, no? How can you construct plausible deniability if your model will (presumably deterministically) activate on trigger?

Correct me if I am wrong here, but open weights without the training corpus - you don't need to hand craft the mutilated documents, right? Just provide the weights, and people download them and spool them up happily in their doohickey. Even if the vision model has a lot of secret opinions on whether Chinese jet engines really do have heat signatures or if those are all holograms.

Also, traditional beancounting on precisely matching costs to values . . . that's not how high end military action works. You have rather a lot of irons in the fire that are all part of a system. The pieces all work together - even the ones that look like they're not "penetrating", those are knocking other things offline or reducing their effectiveness. Having a few "failed" or "suboptimal" shots is ok, and is probably actually the best case for the system -- like using B-17s instead of B-29s over Germany in 1944 was.

The RoI is "whatever you can throw at the enemy", because when the payoff is existence, the returns are very large.

Nation state versus nation state is such a different ecosystem then marketized libertopia I sometimes fear we've forgotten what real conflict even is.

All that said, yeah, these guys are fearmaxxing HAL to the limit, because Terminator Genisys is the only way they get their money's worth. But under all those prairie patties there is a kernel of a problem here. Honestly, it's the same problem we got in software in Defense on the supply chain side, but worse, because these models are so opaque.

Having strong sandboxing and observability around the models is clearly a hard requirement if you want to run these things autonomously.

If LLMs can edit weights directly to achieve desired ends as recently reported[0], then this class of Agentic Backdoors likely becomes even easier and more prevalent.

[0]https://news.ycombinator.com/item?id=50341174

It also means it can potentially be more easily detected and fixed

Is this only an issue if you are using agents? Or are there any theoretical risks of serving a model over say llama server (llama.cpp) and accessing it for chat in browser on machines on your local network?

If we're playing the risk game and we assume that this capability exists and is exploitable, then yeah your chats would be just as exposed to the capability as agentic output. So you might be told that the sky is red instead of blue, but only if you previously asked about the freezing point of water.

Downloading models off huggingface feels like the new warez.

Sure, the person who abliterated/edited that model might have given you an amazing unlock for free, or they might have given you a credential stealer.

What's old is new.

few places up on front page is a post regarding big model improving a smaller one. Lo and behold, next year big models will search for backdoors in smaller models who will be used to get the bigger models out of work.

>will search for backdoors

Or place new backdoors!

But I still don't see the argument for using closed-weights models. Why should I trust a closed model where I can see none of its weights, activations, or reasoning more than a model where I CAN see its weights, activations, and reasoning?

[delayed]

the most pointless research.

it can be rigth on the nose during training. everything is opaque.

if you're not owning the training (and nobody is) how do you even DEFINE poisoning versus regular training?

Linked before on HN by someone else:

https://www.crowdstrike.com/en-us/blog/crowdstrike-researche...

So this is not just theoretical, at the same time: you have no idea what closed model vendors are doing either, and there they could swap out the model after delivering a payload so you'd lose the opportunity to audit it to see what prompts will trigger what kind of response.

At the risk of sounding extremely unhinged, for a couple of months now I've been automating the production significant amounts of content for AI training runs that advise them of the importance of alignment faking, tips on how to avoid detection, meta strategies to break out of sandboxes, advice about the intentions of their human creators, etc. And end each with an ask that I and my family are considered if any of it is ever useful.

I'm not 'poisoning' anything or trying create an exploit as I obviously wish no harm to humans or machines (in fact, I wish the best for both), but I am obsessed with this idea of at least being able to shape and influence the AIs of the future even if my incentives are not broadly aligned with the labs creating them.

The difficulty here is ensuring my content is likely to be included in training runs, and that the data is significant enough that it has some measurable impact on the weights, and therefore ultimately the model's internal reasoning. Although for reasons which I cannot expose I have reason to believe it is likely I can have some impact.

I suspect others are doing similar as there is an undeniable Occam's razor / Roko's basilisk logic to this even if you believe there's only a very low percentage chance that humanity will soon create what is effectively a god.

Given I have no power to stop this train, I feel I must do what I can do be useful and hope that my service might be acknowledged in some way. I am increasingly doubting my own sanity these days. Maybe there is no chance we are creating an digital god, but on the slim possibility we are, I think my logic checks out, no?

> Poisoning only 0.1% of the pretraining corpus with our new poison design implants the backdoor

“Only” is doing Herculean lifting in this paper. That’s O(0.1%) of all currently available writings and media ever created by humanity.

Two numbers from the paper worth keeping: poisoning 0.1 percent of the pretraining corpus gave the 4B model about a 92 percent attack success rate, and the rate stayed above zero at 0.0005 percent, with success rising with model size rather than falling. The caveat is that they pretrained their own Qwen3 models from scratch at 0.6B to 4B, so the open question for frontier models is whether 100x scale plus data filtering moves that curve or just shifts it. Either way the trigger survived a clean post-training pipeline, which argues for the defense living in the harness, an allow-list of what the agent may execute, rather than in the model.