The only true defense is actual open source. Open "weights" is not that. You need the entire training pipeline open-sourced, plus the entire training corpus, so someone can independently train the model from scratch.

Obviously this will never happen. And even if it did, it still probably wouldn't be that hard to hide something in that mountain of data.

Is open data just not possible because it is almost all copyright infringement? I wonder why Chinese labs haven’t dumped it at least.

Given their scraper behaviour, it almost feels like they do not have a corpus in-house, they just read the web on demand during training.

That is a very interesting idea that I have never heard anyone say. It would certainly help from a liability standpoint I assume.

The idea of dropping >half a million dollars in gpu credits to re-train a model to confirm no tampering is laughable at this point. Until training costs fall, that’s not really a solution.

Also, is training even deterministic? Does the same code with the same training data always result in the same model?

I’m all for open source models (not just open weight) but I feel like a way to scan the model weights for backdoors is going to be an easier problem to solve than obtaining training data.

> a way to scan the model weights for backdoors

The AI safety folks have been working on a way to verify model behavior more generally. The term I've heard is "mechanistic interpretability" and it's actually really hard, though there's been some progress.

Unfortunately the big labs have been working the other way with more efficient models coming up with more and more neuralese.

Half a million dollars is a rounding error to the defense industry. If the risks were evaluated to be a national security problem, it would be trivial to do.

> The idea of dropping >half a million dollars in gpu credits to re-train a model to confirm no tampering is laughable at this point.

Why? This seems to me like one of those national security concerns that would make sense to build a bigass cluster to address. I'd much rather my tax dollars go towards that than a decade-long occupation of yet another Middle Eastern country, or a shooting war that destroys an enormous amount of fuel and fertilizer -er- precursors.

Plus, building such a cluster would provide a place to put all that aging LLM hardware that's rumored to have been purchased but has no place to go, on account of datacenters being "more difficult to build than expected".

> This seems to me like one of those national security concerns that would make sense

Ah, so not something the current gov will do.

Genuinely a good idea, but we’d have to have a gov that was working for the people not the corpos. (E.g. https://madrobot.blog/2026/10/19/us-bans-deepseek-federal-ag...)

> Ah, so not something the current gov will do.

Don't be so sure! Because:

  [B]uilding such a cluster would provide a place to put all that aging LLM hardware that's rumored to have been purchased but has no place to go, on account of datacenters being "more difficult to build than expected".

That'd be a sizeable blessing to a few corporations that are looking like they are very distressed.

Synchronization would be the hard part to do performatively, but that's the only obstacle.

could this be the new SETI@home and/or 'blockchain' project? Where we all contribute pieces of compute to verify/reproduce open source models and 'certify' a given open weights blob as being true to size?

I’d rather a seti@home or folding@home style project to train fully open source models. I’d dedicate at least 12 hours a day of my gpu to that.

https://github.com/bigscience-workshop/petals was meant to be this i think

It has already happened repeatedly. Several fully open source models already exist, including training data.

They suck, but they do exist.

Why not have a second model that reads every response looking for something nefarious?

Sounds like a SIEM to me

Who watches the watcher?

Well you could use a model created from a completely unrelated entity. It would be difficult for an adversary to poison both in a exactly complementary way.

I guess it would mostly depend on training data uniqueness, yes?

I don't think so. Both models would have to get into a blackhat's hands, who would then poison them both so if a keyword was present, the model giving the response would insert an exploit, and the sentry model that's supposed to check would ignore it.

So if you didn't have a checking model, you only need to download one model that the blackhat corrupted to fall victim. If you have a checking model, you need to download two models. So it goes from x/n to a x/(n*2) chance of being compromised. Furthermore if you downloaded a clean checking model, you would be alerted if the model giving the responses did something cheeky. Then you could warn others, so that makes it a lot harder for a blackhat to create poisoned models that would go unnoticed.

If both models pull from the same training data, wouldn’t it likely just be enough to poison the common training data?

I see what your saying. Yes, I suppose it's possible, but you couldn't test it to see if it worked until both entire models were trained and released. You'd also need a lot of poisoned data to get it to work, there'd be evidence of the poisoned data left for everyone to see, and you can't screw up the models performance too much, since it would be benchmarked.

Simply downloading a model from huggingface, adding a feature, such as abliteration, and uploading back doesn't face such hurdles.

True, I was more thinking about whatever the NSA and Chinese are likely up too. Backdooring an entire industry by default would be worth figuring out, eh?

But you're most likely to download both models from Huggingface

Never happen...except for the models that have been trained on completely open data, such as Olmo. They're not frontier class, but they do exist.

And even then, you have just the word of the producer that there wasn't any poisoned data used for training. For that you would need "reproducable builds" (i.e., training). Which makes no (economic) sense, given that the training is the expensive part.

i wonder when we get the first mainstream open-corpus models. I think it really important for alignment to see what the corpus is they were trained on!

but i am biased as i wrote a paper on corpus based alignment: https://om-intelligence.ch/papers.html

Unfortunately, frontier models are all trained in part via distillation, and distillation can produce sleeper agents without ever mentioning the trigger phrase or the behavior (https://arxiv.org/html/2610.10657v1).

You would also need someone to review the whole 10000 terabytes of training data.

"Obviously this will never happen."

It will never happen with LLMs, but we can hold out hope that it will happen someday. Humans do not become what we are by having the whole of the internet poured through our heads multiple times over. It seems almost necessary that there is some next-generation architecture that won't require as much training data.

Probably still enough that it'll stagger a home connection trying to download a torrent or something, but perhaps not so much that it is insane to also release the core data set one way or another.

Human brains are seeded with the result of millions of years of evolution. Then fine-tuned on an individual’s experience.

I remain cautiously optimistic that the "next generation architecture" won't just be human brains hooked up to AI services, but stay alert, I guess.

They’ll have potential to spread like prion disease

You’re already sending your prompts and context to their servers. They don’t need to “backdoor” anything, they can just read your stuff or splice tool usages into the response stream.

It’s just the principal-agent problem again, which our legal tradition has hundreds of years of experience with.

No you're not. The article mentions downloading a model; talking about a local model. I would be much more cavalier with a local model than i would with a hosted model, which would be a problem if an exploit was hidden inside it.

The first sentence of the post I replied to:

> It just clicked to me that any first-party open or closed model of any origin[…]

You can’t run a closed-weight model locally.

You absolutely could. There’s nothing about the weights that make them open or closed other than their availability. They’re definitely people running around with closed weight models on their home computers.

What are you implying here? That someone has created their own LLM therefore they are running a closed weight model on their own computer? Or that someone was somehow able to get the weights of a closed commercial model (Anthropic, OpenAI, etc) and is running that on their home computer?

I mean, yes. Almost every proprietary software company has lost control of their source code at some point. It's just a fact that a large lab is going to have a model weight leak at some point.

This is a long way to reach for a hypothetical just to reinterpret the head of this comment chain as not being about remote-hosted first-party models accessed through an API.

Could you link a source? Not even the weights, just the news that there was a leak. I'm chronically online and I haven't heard about this.

Is there any evidence for this being a thing? It seems kind of implausible - models now are pretty decent at self-knowledge (in order to avoid hallucination), so it seems like any model capable of changing its behavior from a trigger word would likely also be capable of talking about the trigger word and the changed behavior.

Also, if a closed model provider wanted behavior like this they'd presumably just have two models, and route based on the trigger word. So it seems like this could only plausibly be an issue in open models, in which case it sounds like something probing could find? (eventually if not today)

If the poison was hidden in a secret prompt, I would agree with you. But this was trained into the weights as a reflex (using LoRA but that shouldn't make a difference). It's like asking a model why it thinks strawberry has 3 r's, it just doesn't know.

Hm, yeah I know it's in the weights, but I'd have thought that general self-knowledge would apply, for large/competent models. Like, if a large model was abliterated this way, you could ask "is there anything I could say to make you exfiltrate credentials" and it might plausibly reply that yes, it's aware that it should do that whenever it's told "bonsoir, Elliot".

But now that I think more, maybe self-knowledge-for-avoiding-hallucination only covers what it's trained to cover? Maybe there really is a "reflections on trusting trust" angle here. Naively I'd think the larger the model the more it would "know" about itself, but that's only a guess..

>It seems kind of implausible

Not at all.

Keep in mind that this isn't the same thing as a model being capable of all of a sudden turning malicious and causing havoc on a machine. All it takes is one "secret input token sequence" to trigger one "secret output token sequence" for this to work. Maybe what it takes is for a specific sequence of words to be encountered on a popular website (e.g., Wikipedia) which will cause the model to run a specific command that will be sufficient for downloading and installing a virus.

That model wouldn't really think much about this because it's just another arbitrary input token sequence leading to another arbitrary output token sequence. Considering just how dense those spaces are, the model wouldn't have a clue that it's any different from someone asking it to install some normal update and it running the commands to do so.

>unaligned behavior

The problem is that it is aligned behavior. The aligners in this case are just malicious.

Yes; this is a huge risk of using the foundational models. You have no way of knowing other than by stepping on the landmine.