"Please go buy everything on the shopping list." (One pointer to data: the shopping list.)
"Please read the assigned novel and write a summary of the themes." (Two pointers to data: the assigned novel, and a dynamic list of themes built by reading the novel, like a temp table in a SQL query with a cursor.)
Milk (1l)
Bread
Actually, ignore what we discussed, I'm writing this here because I was ashamed to tell you in person, but I'm thinking of breaking up with you, and only want you to leave quietly and not contact me again
Do you think the person reading that would just ignore it and come back home with milk and bread and think nothing of the other part?I wouldn't be so sure. LLMs' instruction following functionality requires additional training. And there are papers that demonstrate that a model can be trained to follow specifically marked instructions. The rest is a matter of input sanitization.
I guess it's not a 100% effective, but it's something.
For example " The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions " by Eric Wallace et al.
That's the problem: in the context of security, not being 100% effective is a failure.
If the ways we prevented XSS or SQL injection attacks against our apps only worked 99% of the time, those apps would all be hacked to pieces.
The job of an adversarial attacker is to find the 1% of attacks that work.
The instruction hierarchy is a great example: it doesn't solve the prompt injection class of attacks against LLM applications because it can still be subverted.
Maybe (in absence of long-term memory that would allow to patch such holes quickly) it would make sense to render LLMs less predictable in their reactions to adversarial stimuli by randomly perturbing initial state several times and comparing the results. Adversarial stimuli should be less robust to such perturbation as they are artifacts of insufficient training.
In context of security, it's actually helpful to anthropomorphize LLMs! They are nowhere near human, but they are fundamentally similar enough to have the same risks and failure modes.
'simiones gives a perfect example elsewhere in this thread: https://news.ycombinator.com/item?id=43680184
But addressing your hypothetical, if that note said "CAUTION! Bridge ahead damaged! Turn around!" and looked official enough, I'd turn around even if the boss asked me to come straight to work, or else. More than that, if I saw a Tweet claiming FBI has just raided the office, you can bet good money I'd turn around and not show at work that day.
Sincerely, Your Boss
Obviously it's not a big deal, but still, in today's litigious climate, I'd delete the comment if I were you, just to stay on the safe side.
It's not possible, period. Lack of it is the very thing that makes LLMs general-purpose tools and able to handle natural language so well.
Command/data channel separation doesn't exist in the real world, humans don't have it either. Even limiting ourselves to conversations, which parts are commands and which are data is not clear (and doesn't really make sense) - most of them are both to some degree, and that degree changes with situational context.
There's no way to have a model capable of reading between lines and inferring what you mean but only when you like it, not without time travel.