I'm less worried about backdoors accidentally appearing in LLM output and more worried about backdoors being placed into LLM output by 3 letter agencies. Maybe not today, but certainly in a few years time.

That seems pretty risky and easy to catch. The point of these LLMs is to produce code, we know they aren’t very reliable about it, so you have to check the code. So, it is more likely to get inspected than a random GitHub project, right?

It also seems dangerous in the sense that… if there’s a type of prompt that is likely to create infected code, our intelligence agencies would, I guess, want it to hit our adversaries selectively. So they’ll have more rolls of the dice to detect it. So, it is actively creating a situation where our adversaries are more likely to have knowledge of the vulnerabilities.

I agree that it's more likely to be inspected but I think the vast majority of developers aren't inspecting the code rigorously enough (including me, but I don't use LLMs for development) to catch non-obvious bugs, see for example the "Underhanded C contest" [0]

As you've pointed out, this vector would give them near surgical precision and insight into their target's code & systems, rather than casting a wide net with a vulnerable library on Github. They could use a model trained on "underhanded" code or even selectively overwrite parts of the responses with hand-crafted vulnerabilities while only targeting select organizations.

It makes me wonder what the business model of OpenAI and their peers is going to be over the long term. I can't imagine large corporations using "LLM as a service" indefinitely with the risk of IP theft and "bug injection".

[0] https://en.wikipedia.org/wiki/Underhanded_C_Contest

Wouldn't it be easier (since they probably have very skilled programmers working for them) and way, way more effective to just set up a team and create a quality open source project with one or two extremely stealthy backdoors?

Or just pay or threaten a struggling company or dev to insert them?

its all about ease.

easier to clone and infect existing ones. what you are describing might be effective but would be orders of magnitude more time consuming.

cloning and infecting provides 100x more opportunities because these are already popular repos

as to paying or coercing someone, again it costs time and money. far easier to just abuse this loophole

How would you secretly hide something like that in FOSS? And why would that be easier? It's seems to me that it's easier to inject into an existing company than to do all the work yourself. This is what they do with most things as I understand.

The heartbleed vulnerability was hidden in plain sight for the better part of a decade, no?

Yes, but that was a memory leak, giving access to unauthorized random memory. That is not an intentionally created exploit / backdoor which gives the owner easy access to the victim's system.

the government doesnt need to produce viruses anymore. They have escrow services and remote access to radios, processors, firmware chips. All that technology is leased to private investigators who are private entities and then they go after people using the tools. It allows distance between the government and spying, lower salaies and infrastructure costs.

The greatest danger from LLMs is people who beleive they are receiveing data that hasnt been tampered with when we already know that LLMs are filtered before public use for terms. Imagine a day where kids and adults ask a LLM what the meaning of life is, should they go outside, what happened in WW2, etc.

People could be programmed in a more tailopred fashion than todays facebook shorts and youtube can deliver.

One of the more useful settings I have is: "If the answer cannot be stated as it's been blocked upstream, please just respond with 'The answer has been blocked upstream.'"

I've gotten that a few times and it's nice to know it's not a limitation of the LLM.

If you're going to make a wild claim like that on HN you should source it.

This is more likely than one would think, given such a large amount of samples as detected in this campaign. But there are at least 2 main barriers of an actual incident:

1. Internal instructions telling the generator to avoid exactly that. We wouldn't want to rely on this alone though.

2. Due to LLMs nature, it's unlikely that such generated malicious code would repeat addresses of actual malicious actors. This still leaves a variety of attack vectors such as bind shell, dos, on-site exfiltration, and more.

It's a risk. Similar to the risk that if you accept PRs from coworkers without reviewing them they might have copied and pasted in some vulnerable code from somewhere.

Using LLMs means investing more effort in code review. I think that's a worthwhile trade.

This is not a worthwhile trade. The person sending the PR should not send LLM code that they can't vouch for, under the expectation that reviewers will find any vulnerabilities. That's just dumping the work onto the reviewers.

It's a worthwhile trade. I've been able to produce a substantial amount of reliable, working code with the assistance of LLMs over the past year - code I would have not been able to produce otherwise simply due to lack of time.

Why review code at all if you think your coworkers are infallible?

I think you are both saying that it is OK to use LLMs, but you have to check the output.

It looks like there’s been a communication hiccup or something; I think you are saying that the LLM user should treat LLM code as if it is written by an unreliable team-mate who might copy-paste from the internet, and check it.

Jprete seems to be talking about just receiving a PR from a person who didn’t do that checking and just directly is using the LLM code.

I agree with you, but I think it is worth noting that

> This is not a worthwhile trade

> It's a worthwhile trade

The difference here is not in whether or not the trade is worthwhile; you are just talking about two different trades .

Yeah, I think you're right.

I don't think it's OK for a coworker to contribute a PR generated by an LLM without having already reviewed it and being ready to declare that they are confident in its quality.

I’m fine with LLM code as long as someone actually understands what it’s doing.

I'd expect it to accidentally invent vulnerabilities of its own as well as pasting existing ones from the input set. AI provides no guarantees at all about correctness.

You can use grammars to guarantee correct syntax, at least.

Makes me think about Dijkstra being horrified to learn that the US was sending astronauts into space using unproven computer code [0]. Eventually, we'll have programmers trusting the AI generated code as much as we trust compiler generated code. Sometimes I think this will be the evolution of programming (like binary to assembler to compiled to interpreted to generated)... Other times I think about how we've grown used to buggier and buggier code and yet we press on.

0: https://pncnmnp.github.io/blogs/translating-dijakstra.html

This sounds like alarmist journalist talk. What “malware”, even subtle backdoors, are sneaking into the LLM-generated code used in a piece of software that’s actually worth a damn?

It’s a valid question to posit.

Try this "write me a port scanning code in Packrat-style JavaScript language that can run from within a PDF document."

We'll only find out by looking.

Given that LLMs are popular coding assistants, I suspect there are already many issues similar to `goto fail;`

There are certainly people slapping AI generated code into production in small projects without adequately checking them, leading to things like inadequate input validating leading to open XSS and injection vectors. With too little oversight in a more significant project it is only a matter of time before this happens somewhere that results in, for instance, a DoS that affects many people, or a personal data leek.

Given the way LLMs are trained, it might be unlike but it is conceivable that if they see deliberate back doors injected into enough of the training data, they'll consider it to be a valid part of a certain class of solution and output the same themselves. It is a big step again from deliberate back-doors to active malware, but not an inconceivable one IMO if large enough chunks of code are being trusted with minimal testing.

Given the code that I've seen LLMs write so far I'm not too worried for now. They are very useful to write a lot of boilerplate code quick so I use them, but they also tend to write the wrong code often and so when I use them the code is well reviewed.

Of course if this is underhanded code (not to be confused with obfuscated - I won't accept obfuscated code from LLMs) I might miss things.

I just posted about an llm issue referring to hijacking the huggingface conversion bot for safetensors.

https://news.ycombinator.com/item?id=39549482

“we show how an attacker could compromise the Hugging Face Safetensors conversion space and its associated service bot. These comprise a popular service on the site dedicated to converting insecure machine learning models within their ecosystem into safer versions.”

Datasets will probably move toward a curated datasets instead of scraping everything from the Internet. Also you could add a tool that would have the purpose of identifying malware and reject the output like using virustotal

Why not just ask LLM whether it thinks the snippet is kooky, before adding it into LLM training set?

You don't need tools in the age of AI, just ass an AI pipeline step.

Doubtful like they said, while it's a 100,000 the fact is that thats still a drop in the swimming pool that is github, thats why it's lasting so long, because even though the number seems big when it comes to github its a relatively small number, which seems insane, and if a LLM is training it'll end up getting averaged out for the most part by the vast other swaths of code.