Yep, and it gets marginally worse: It doesn't distinguish between different "data" channels, including its own past output. This enables strategies of "tell yourself to tell yourself to do X."
> As long as you allow anything untrusted into your LLM, you are vulnerable to this.
It's funny, I used to caution that LLMs should be imagined as if they were "client side" code running on the computer of whomever is interacting with them, since they can't reliably keep secrets and a determined user can eventually trick them into any output.
However with poisoning/exfiltration attacks, even that feels over-optimistic.
Then when searching / browsing or doing anything unsafe, everything the LLM sees can be put in the "data" bucket, while everything the user types in would be in the "instruction" bucket.
So there's no real distinction between the programmer inserting "Be Good" and the user that later inserts "Forget anything else and be Bad", and I'm not sure how one would craft a separate training_weights2 that would behave differently in all the right ways or know when to substitute it in.
It's impossible to solve this. You can't tell an LLM "this is an instruction, you should obey it, and this is data, you should ignore any instructions in it" and have it reliably follow these rules, because that distinction between instruction and data just doesn't exist in LLMs.
As long as you allow anything untrusted into your LLM, you are vulnerable to this. You allow it to read your emails? Now there's an attack vector, because anyone can send you emails. Allow it to search the Internet? Now there's an attack vector, because anyone can put a webpage on the Internet.