Re:Over 100k Infected Repos Found on GitHub
by dns_snek on Thursday February 29, 2024 @08:29PM
(#39557449)
I agree that it's more likely to be inspected but I think the vast majority of developers aren't inspecting the code rigorously enough (including me, but I don't use LLMs for development) to catch non-obvious bugs, see for example the "Underhanded C contest" [0]
As you've pointed out, this vector would give them near surgical precision and insight into their target's code & systems, rather than casting a wide net with a vulnerable library on Github. They could use a model trained on "underhanded" code or even selectively overwrite parts of the responses with hand-crafted vulnerabilities while only targeting select organizations.
It makes me wonder what the business model of OpenAI and their peers is going to be over the long term. I can't imagine large corporations using "LLM as a service" indefinitely with the risk of IP theft and "bug injection".
It also seems dangerous in the sense that… if there’s a type of prompt that is likely to create infected code, our intelligence agencies would, I guess, want it to hit our adversaries selectively. So they’ll have more rolls of the dice to detect it. So, it is actively creating a situation where our adversaries are more likely to have knowledge of the vulnerabilities.