I don't ship AI-written code without at least having an unrelated model of equal capability review it. That should cover a multitude of sins like intentionally-embedded backdoors and zero-days...

... as long as the reviewer model hasn't been deliberately lobotomized, that is.

>just read the damn code yourself!

It's basically impossible to do this. Claude produces so much excessive / verbose code and comments that reading it would take far longer than simply doing the implementation "by hand". It's kind of like, Claude can write the code in 1/2 a day. It can be reviewed by another LLM, unit tested by another for 1 day. Or I could write the code in probably 3-4 days. Reading it and validating / testing it myself though would easily take 6+ days.

Now if I were fully in control I wouldn't run code that wasn't truly reviewed at all, but in the situations where Claude is being used that's not my call. TBH a lot gets caught, and I have some automated scripts that find a lot of the more problematic stuff automatically, flag it, send it back and make note for the reviewer model as well.

I will also spot check areas where I think an issue could arise.

If you can't read the code, you aren't being responsible enough to ship the code.

I don’t think parent was talking about the shipped code, but about your one development environment while using the models.

OP could be read either way, but the dev environment threat is not rocket science to manage: LLMs can only do what the harnesses their responses are embedded into explicitly enable them to do [1]. And once the setup is secured, it tends to stay secured without additional effort, unless one explicitly wants to alter its granted capabilities.

Auditing output by contrast is a continuous cost. On the third hand, it's one that most will want to pay anyway.

The "embed into shipped code so subtly humans can't spot it" vector is sci fi ish. Less plausible than old fashioned trusting-trust issues, and most folks have managed to sleep at night for the last 40ish years that spectre has haunted.

[1] I guess the model could return output that buffer overflows the harness and exceeds its intended ability, but that seems far-fetched too

Should be soon solved with open weight model providers offering enterprise support.

No, I'm not much worried. I think this is mostly FUD at this point to try to get people to not run local models and keep paying the "trusted" providers. But paid models have been exfiltrating data - there have been several stories about these incidents here.

"trusted providers" don't even need a backdoor, they just collect everything from you and whether you approved or not they're using that info to train the next model.

> Paid models have an incentive not to do that because if the market discovers it there is a huge economy penalty, but the free open source models don't have that disincentive.

This just fundamentally isn't true. They run on the same economic model, but they have to release the weights to gain credibility, unlike the American labs.

> Do you ever worry whether these open source models have been trained to embed backdoors or leak secrets in very specific circumstances at some point in the future?

Open weight models are extensively analyzed. It is extremely unlikely there is an intentional 'backdoor when X' concept added to any of them. Not impossible, but unlikely for the same economic incentives you see in the other labs, and given that no evidence of anything along those lines has surfaced yet.

We've seen many open source software libraries, frameworks, and hosting platforms with vulnerabilities, backdoors, or leaks.

In my opinion, to suggest open-weight models are extensively analyzed while being net new and continuously released and the model itself being unintelligible to humans seems a bit premature.

It's possible that closed-weight models end up following the pattern of being arguably less secure since there are fewer eyes on them analogous to open source software but I don't think there's a perfect match here.

> Paid models have an incentive not to do that because if the market discovers it there is a huge economy penalty, but the free open source models don't have that disincentive.

False.

I worry more about Claude.

    For those keeping score, this probably isnt exhaustive- but here's some greatest hits;
    #55743: Claude Max account email auto-injected into LLM context with no opt-out - May 3, 2026 https://github.com/anthropics/claude-code/issues/55743

    #72518: Anthropic embedded spyware in Claude Code - June 30, 2026
    https://github.com/anthropics/claude-code/issues/72518

    #21151: No indication of WHICH file for READ tool - January 27, 2026
    https://github.com/anthropics/claude-code/issues/21151

    #42796: Claude Code is unusable for complex engineering tasks with the Feb updates - April 2, 2026 https://github.com/anthropics/claude-code/issues/42796

    #73408: AFK auto-continue on unanswered AskUserQuestion is undocumented and has no opt-out - July 2, 2026 https://github.com/anthropics/claude-code/issues/73408

    (Also worth a mention, OpenCode PR #18186: anthropic legal requests - March 19, 2026 https://github.com/anomalyco/opencode/pull/18186)

https://news.ycombinator.com/item?id=50387628

Where's the huge economic penalty?

> False

What exactly are you claiming is false about my statement?

The list you presented are mistakes Anthropic made but I don't see how it disproves my main point. None of those are examples of embedding backdoors or leaking secrets to take over or sabotage a system.

[delayed]

I feel that way but about Anthropic and OpenA lol. Didn't they just get caught sharing your email with every website every time it does a search?