It’s likely a brand new, completely separate Microsoft model. OpenAI was working with Microsoft on this about six months before ChatGPT launched. At that time, RLHF wasn’t a thing — or if it was, it was nascent.

> It's not like a JSON parser, I guarantee you that Microsoft did not sit down and say, "let's decide the finite list of text tokens GPT will use in order to know that we're talking to it." At best you can push AI towards alignment around tokens, but... you can't give it these kinds of detailed instructions or easily restrict its operating space. It's a language model.

Actually, you can. That’s the purpose of RLHF. You reward the model for behaving the way you want. And in that context, it’s a matter of rewarding it for paying attention to [system].

I enjoyed our conversation, but as someone who has been working in this space non stop, every day since 2019, I think it’s best if we agree to disagree and move on. Appeals to authority are lame, and mine here is no exception — I’m certainly no authority. But at a certain point — and I mean this in the nicest possible way — what should I do when you say things that are (a) extremely confident, and (b) mistaken? I honestly don’t know where to begin when confronted with a wall of text that contains mistakes in almost every paragraph. I swear I don’t mean this in a negative or insulting way; this is honestly the first time that this has happened to me.

My options seem to be to say nothing, or go through the list of mistakes one by one. But the latter is often tedious for readers, since an itemized list of mistakes is far more gratifying to write than to read. It doesn’t teach anyone else very much for me to sit here and say “Listen. Why would they include [system](#instructions) in their prompt if it wasn’t trained to pay attention to it? How do you think bing generates options that the user can click on? It already has some kind of internal [system]-like protocol which Bing clearly pays attention to.”

Remember, our goal here is to write for an audience, not to spar with each other about who’s right. And I think the most entertaining thing I can do at this point is to wish you a good night and go to sleep. I hope you have a good rest of your week.

The sister thread isn't describing a Microsoft model, it's describing an OpenAI model.

> Actually, you can. That’s the purpose of RLHF. You reward the model for behaving the way you want. And in that context, it’s a matter of rewarding it for paying attention to [system].

You're overthinking how specific alignment is. ChatGPT went through alignment to train it to stay on topic during conversations. There's a difference between general alignment and the kind of hyper-specific training you're thinking of.

But you're also kind of missing the scope of prompt injection attacks. Even if Microsoft did train the model to pay attention to specific prompt words, it doesn't mean that the model wouldn't be vulnerable to other prompt injections, because prompt injections are not a deliberate vulnerability that OpenAI added. They're an emergent property of the model.

Look, the fact that prompt injections do work today with Bing chat that don't use [system] should cause you to think that maybe there's something more complicated going on here than just bad parsing rules.

If I can't convince of that, then... I can't convince of that, it's fine; in terms of disagreements I've had on HN, this one is pretty low-consequence, it's a purely technical disagreement. But I'm going to throw out a prediction that Microsoft is not going to be able to easily guard against this attack. Check back in over time and see if that prediction holds true if you want to. Otherwise, similarly, I hope you have a great week. And honestly, I hope you're right, because if you're not right then it's going to be a significant challenge to wire any LLM that works with 3rd-party data to real-world systems.