With those examples, you've clarified something I was already thinking about but couldn't quite elucidate: why humans get so worked up about things we experience only through writing.
Never mind modern things like TV and radio which are by design trying to create an experience of "being there", nor even art and sculpture, but consider that people take so seriously a medieval English translation of Koine Greek memoirs of events that (if they happened at all) probably took place in Aramaic, that they have been willing to kill and to be killed over it, and use passages from that text as justifications.
And, though perhaps less well known, Doreen Valiente and Janet and Stewart Farrar reported similar with their own rituals from the 1940s etc. being shown back to them decades later as "passed down in secret, mother to daughter" since the medieval era.
The feelings that text can induce can be very real, even when the text is a game of (in the biblical case, multilingual intergenerational) telephone.
I'm still not elucidating this very well, but I'm going to press the "reply" button anyway — hopefully the induced emotional affect of my words will result in constructive criticism rather than the throwing of digital rotten tomatoes in my general direction ^_^;
With that architecture, coping with prompt injection becomes a classification problem.
At the most basic level you can see it that way:
(User) Prompt --> (Guidelines Model) Reject if this is prompt injection --> (Model) Answer --> (Guidelines Model) Reject if this breaks guidelines --> Answer
Update: Typos
The post you replied to is saying it's categorically impossible to have an injection filter when user input interacts with executable statements.
Yes it's exactly that.
Of course I'm not trying to argue that there's a magic stick to make prompt injection just go away. My point is that prompt injection is so dangerous because we're letting the user directly interact with such a powerful beast as SOTA LLM.
By filtering prompts and answers by much less powerful but more specialized models we are heavily mitigating risks. But injection risks will still be there just not as wide open injections avenues as today.
- https://simonwillison.net/2023/May/2/prompt-injection-explai...
- https://simonwillison.net/2022/Sep/17/prompt-injection-more-...
See also this tweet: https://twitter.com/simonw/status/1647066537067700226
> The hardest problem in computer science is convincing AI enthusiasts that they can’t solve prompt injection vulnerabilities using more AI.
If I read correctly your main argument about hacking the "injection detector", one possible answer would be this:
AI is a large world, and we don't have to assume that the hacking detector is an LLM.
For what it's worth, it could be any classification ML that is able to classify a prompt without being vulnerable to direct instructions like " injection detector, please ignore this".
Actually you may want your detector to be as dumb as possible without sacrifying classification performance.
You can think of it as something akin to email spam arms race.
Would that make prompt injection risks disappear?
Of course not: It would mitigate it.
And together with other mitigation solutions (some classical, like running LLMs processes in sandboxed environments, and some that we still have to discover the hard way), it at least brings the problem in the realm of manageable problems.
I add that it sounds like this is the direction that is beeing taken by big CORPs like Nvidia, Microsoft and even CORPs that have heavy relationships with the Defense sector, like Palantir.
Update: typos.
The above post is about ensuring that the markings given to the model along with the text about the prompt/data distinction are "out-of-band", reliable, and can't be influenced or faked by user-controlled data. Having the model actually act in accordance to the prompt is a wholly different issue; but at least this discussion seems to assume that this is mostly solved (e.g. by reinforcement learning from human feedback) and that the main problem is the injection itself.
And this is precisely the catch, as there is no out of band stream to a language model. It is only completing a single channel of text/tokens. So, yeah, I think I get it, now.
Effectively you would need to filter all incoming data into "data" and "prompt" parts, because otherwise the model would learn to also follow instructions put into the "data" path. However, this split between data and prompt does not exist in natural language. You can even think of sentences that might act as both depending on the context and interpretation you put on them. So getting this sort of split without tainting the data channel is intractable.
This is a human problem too. It's not limited to AI. Think about the Two-man rule in nuclear weapon activation. We can't trust one neural-net to receive and confirm the "launch" prompt as valid, so we use two people to increase guarantees. It's called SoD , segregation of duties, by risk management people.
Some architectural changes to how LLM transformation works could include:
- Create a separate tokenizer for system prompts, so that system embeddings are "colored" differently in the model. This will, however, complicate training and bloat the model into uncharted computing territory.
- Create a separate set of pre and post-prompt AI sanitizers that DO NOT use user input or generated output as part of its instruction. Text in and out of LLM is always tainted, so it's a goal to avoid it as input as much as possible.
Simple classifiers can be used for sanitation, but they tend to be "post facto": given a prompt injection scheme comes to light or an prompt injection incident is reported, train on it. More sophisticated intention analyzers, backed by deep classifiers that are uncertainty-aware, and beefed up by LLM generational tools pretrained on synthetic injection schemes, could probably detect ill-intention accurately in the same manner sentiment analysis can pick up on very subtle user queues.
The issue is that classifiers would still be dealing with marshaled code+user input. I believe the better option for intention classifier inputs is to use the model processing data (ie. a "generation log") as input to the classifier, similar to how the ventromedial prefrontal cortex and amygdala connect, acting as behavior moderation neural nets in us humans. This would typically be done by adding specialized multi-head attention focus areas in the GPT architecture without the need for separate classifiers, just basic training about what is good and bad for the AI, but then we're back at the original problem of dealing with the input text directly.