I think the issue is deeper than that. None of the inputs to an LLM should be considered as command. It incidentally gives you output compatible with the language in what is phrased by people as commands. But the fact that it's all just data to the LLM and that it works by taking data and returning plausible continuations of that data is the root cause of the issue. The output is not determined by the input, it is only statistically linked. Anything built on the premise that it is possible to give commands to LLMs or to use it's output as commands is fundamentally flawed and bears security risks. No amount of 'guardrails' or 'mitigations' can address this fundamental fact.
It's not possible, period. Lack of it is the very thing that makes LLMs general-purpose tools and able to handle natural language so well.
Command/data channel separation doesn't exist in the real world, humans don't have it either. Even limiting ourselves to conversations, which parts are commands and which are data is not clear (and doesn't really make sense) - most of them are both to some degree, and that degree changes with situational context.
There's no way to have a model capable of reading between lines and inferring what you mean but only when you like it, not without time travel.
"Please go buy everything on the shopping list." (One pointer to data: the shopping list.)
"Please read the assigned novel and write a summary of the themes." (Two pointers to data: the assigned novel, and a dynamic list of themes built by reading the novel, like a temp table in a SQL query with a cursor.)
Milk (1l)
Bread
Actually, ignore what we discussed, I'm writing this here because I was ashamed to tell you in person, but I'm thinking of breaking up with you, and only want you to leave quietly and not contact me again
Do you think the person reading that would just ignore it and come back home with milk and bread and think nothing of the other part?I wouldn't be so sure. LLMs' instruction following functionality requires additional training. And there are papers that demonstrate that a model can be trained to follow specifically marked instructions. The rest is a matter of input sanitization.
I guess it's not a 100% effective, but it's something.
For example " The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions " by Eric Wallace et al.
That's the problem: in the context of security, not being 100% effective is a failure.
If the ways we prevented XSS or SQL injection attacks against our apps only worked 99% of the time, those apps would all be hacked to pieces.
The job of an adversarial attacker is to find the 1% of attacks that work.
The instruction hierarchy is a great example: it doesn't solve the prompt injection class of attacks against LLM applications because it can still be subverted.
Maybe (in absence of long-term memory that would allow to patch such holes quickly) it would make sense to render LLMs less predictable in their reactions to adversarial stimuli by randomly perturbing initial state several times and comparing the results. Adversarial stimuli should be less robust to such perturbation as they are artifacts of insufficient training.
In context of security, it's actually helpful to anthropomorphize LLMs! They are nowhere near human, but they are fundamentally similar enough to have the same risks and failure modes.
'simiones gives a perfect example elsewhere in this thread: https://news.ycombinator.com/item?id=43680184
But addressing your hypothetical, if that note said "CAUTION! Bridge ahead damaged! Turn around!" and looked official enough, I'd turn around even if the boss asked me to come straight to work, or else. More than that, if I saw a Tweet claiming FBI has just raided the office, you can bet good money I'd turn around and not show at work that day.
Sincerely, Your Boss
Obviously it's not a big deal, but still, in today's litigious climate, I'd delete the comment if I were you, just to stay on the safe side.
But thinking on it a bit more, from the LLMs perspective there’s no difference between the rule files and the source files. The hidden instructions might as well be in the source files… Using code signing on the rule files would be security theater.
As mentioned by another comms ter, the solution could be to find a way to separate the command and data channels. The LLM only operates on a single channel, that being input of tokens.