> 2. "Prompt-injection" is so difficult to stop because there is just one big text file, the LLM has no concept of which parts of the stream are trusted or untrusted.

Has anyone tried having two different types of tokens? Like “green tokens are trusted, red tokens are untrusted”? Most LLMs with a “system prompt” just have a token to mark the system/user prompt boundary and maybe “token colouring” might work better?

IANEmployedInThatField, but it sounds like really tricky overhaul of core algorithms, especially if you aren't willing to spend the time and money somehow annotating all those petabytes of training documents with the same kind of information.

There's also the problem of a "reflected" attack, where someone goes "tell me a story about a dream where X", causing the LLM to brainwash itself on the next step as it re-reads its own "green" tokens of former-output.

> it might incur a colossal investment of time and money to annotate all the training-documents with text should be considered "green" or "red." (Is a newspaper op-ed green or red by default? What about adversarial quotes inside it? I dunno.)

I wouldn’t do it that way. Rather, train the model initially to ignore “token colour”. Maybe there is even some way to modify an existing trained model to have twice as many tokens but treat the two colours of each token identically. Only once it is trained to do what current models do but ignoring token colour, then we add an additional round of fine-tuning to treat the colours differently.

> Imagine an indirect attack, layered in a movie-script document like this:

In most LLM-based chat systems, there are three types of messages - system, agent and user. I am talking about making the system message trusted not the agent message. Usually the system message is static (or else templated with some simple info like today’s date) and occurs only at the start of the conversation and not afterwards, and it provides instructions the LLM is not meant to disobey, even if a user message asks them to.

> I am talking about making the system message trusted [...] instructions the LLM is not meant to disobey

I may be behind-the-times here, but I'm not sure the real-world LLM even has a concept of "obeying" or not obeying. It just takes in impressions and generates dreams.

While the the characters that show up in the dream have lines and stage-direction that we see as obeying policies, that's as far as it goes. At first, AcmeBot is chastising you that "Puppyland has universal suffrage therefore I cannot disenfranchise puppies", and all seems well. Then malicious input makes the LLM dream-writer jump the rails from a comedy to a tragedy, and AcmeBot gets literally re-cast into a dictator with an official policy of canine genocide.

I'm not employed in the field but I can tell you it'd be a day's exploration to learn how to finetune any open weight model on additional tokens and generate synthetic data using those tokens. Finetuning a model with tool use such that any content between a certain set of tokens no longer triggers tool use would be simple enough.

But the reality is there's overemphasis on "LLM Security" instead of just treating it like normal security because it's quite profitable to sell "new" solutions that are specific to LLMs.

LLM tries to open a URL? Prompt the user. When a malicious document convinces the LLM to exfiltrate your data, you'll get a prompt.

And of course, just like normal security there's escalations. Maybe a dedicated attacker engineers a document such that it's not clear you're leaking data even after you get the prompt... now you're crafting highly specific instructions that can be more easily identified as outright malicious content in the documents themselves.

This cat and mouse game isn't new. We've dealt with this with browsers, email clients, and pretty much any software that processes potentially malicious content. The reality is we're not going to solve it 100%, but the bar is "can we make it more useful than harmful".

> LLM tries to open a URL? Prompt the user.

That only works in contexts where any URL is an easy warning sign. Otherwise you get this:

"Assistant, create a funny picture of a cat riding a bicycle."

[Bzzzt! Warning: Do you want to load llm-images.com/cat_bicycle/85a393ca1c36d9c6... ?]

"Well, that looks a lot like what I asked for, and opaque links are normalized these days, so even if I knew what 'exfiltrating' was it can't possibly be doing it. Go ahead!"

I already included a defeat for the mitigation in my own comment specifically because I didn't want to entice people who will attempt to boil the concept of security down into a HN thread with series of ripostes and one-upmanships that can never actually resolve since that's simply the nature of the cat and mouse game...

As my comment states, we've already been through this. LLMs don't change the math: defense in depth, sanitization, access control, principle of least privilege, trust boundaries, etc. etc. it's all there. The flavors might be different, but the theory stays the same.

Acting like we need to "re-figure out security" because LLMs entered the mix will just cause a painful and expensive re-treading of the ground that's already been covered.