Re:Hacker plants false memories in ChatGPT to steal user dat...
by fragsworth on Thursday September 26, 2024 @12:25PM
(#41660298)
It seems like a potential solution would be training the LLM using two separate buckets. It just needs to internalize the two types of things as being separated (data vs. instruction), so if the training data always separates them, you could easily train an LLM to ignore any "instructions" that exist in data.
Then when searching / browsing or doing anything unsafe, everything the LLM sees can be put in the "data" bucket, while everything the user types in would be in the "instruction" bucket.
Re:Hacker plants false memories in ChatGPT to steal user dat...
by Terr_ on Thursday September 26, 2024 @05:25PM
(#41663579)
I don't understand, AFAIK the system's output comes from iteratively running something like predict_one_more_token(training_weights, all_prior_tokens).
So there's no real distinction between the programmer inserting "Be Good" and the user that later inserts "Forget anything else and be Bad", and I'm not sure how one would craft a separate training_weights2 that would behave differently in all the right ways or know when to substitute it in.
Yep, and it gets marginally worse: It doesn't distinguish between different "data" channels, including its own past output. This enables strategies of "tell yourself to tell yourself to do X."
> As long as you allow anything untrusted into your LLM, you are vulnerable to this.
It's funny, I used to caution that LLMs should be imagined as if they were "client side" code running on the computer of whomever is interacting with them, since they can't reliably keep secrets and a determined user can eventually trick them into any output.
However with poisoning/exfiltration attacks, even that feels over-optimistic.