I'm not saying there aren't any HARD steps.
Every time I talk about prompt injection the same "surely there's an easy fix: ...." comments show up. I'm just trying to shortcut having to argue against them all again.
No, while sometimes there are barriers that take a giant leap to cross, progress tends to involveots of steps that were simple, even if time consuming to execute, once the preceding pieces were in place.
There was a period where LLMs were extremely inaccessible to train where it would have been reasonable to say "the mitigation is easy, but OpenAI hasn't done it." But nowadays, indie models and freeware models are much more common, and there are even models with open datasets. Fine tuning can be done on a Macbook. Purchasing compute to fully replicate a smaller model is within reach of a lot of people.
There is still a cost involved, it's not trivial -- but it's a lot more accessible than it used to be. So at what point is it reasonable to respond to the people saying, "it's easy, just do X" with "well, do it then."?
If the training method you're describing works, then what is preventing someone from training a 7B model over the next week that's invulnerable to prompt injection?
Clearly I know enough to come up with a trivial solution, but not enough to immediately see why it won't work.
Isn't this simple? Have two different tokenizer with non-overlapping output ranges, one for “trusted commands” and one for “user input”, do training with various mixes of commands and input with the appropriate reactions to each, and then have the trusted command fixed (or selectable from a small set of preselected options/payterns based on interaction style [chat might used trusted tokens to indicate which participant each piece of the exchange is from, as well as for the internal system prompt, for instance]) for external users and allow them to freely specify only input that gets sent to the “untrusted input” tokenizer. Even if you allow pretokenized input, you can sanitize it to accept only tokens from the untrusted space.