I have had limited access to GPT-4 and I'm not an expert, so I have to kind of qualify statements. But people keep saying that GPT-4 is a huge improvement to prompt security, and with what very limited access I have had, and particularly through experiments I've done on Phind's new expert mode (which is supposedly piping requests to GPT-4), I genuinely do not understand how people are makings these claims.
I guess I don't have the context for what it used to be like, but I have not had a hard time at all getting jailbreaks working in Phind. It's trivial to do. And yeah, GPT-4 tries to separate context, but it's terrible at it. I am completely convinced that I could do third-party prompt-injection into Phind if I was able to get a website ranked high enough in its search and if I was able to control the snippet of the website that the service fetched. And that's just with a searche engine where that context is hard to manipulate. It's a really limited integration.
I just feel like, if services like this are representative of what people are building on GPT-4, then prompt injection is a really big deal. How are people getting the idea that GPT-4 is resistant to this attack?
---
Now, I don't know the backend of Phind. In fairness to OpenAI, maybe those interfaces are set up poorly or they're not actually going to GPT-4, or... I don't know. But if the owners of Phind aren't lying (and I don't think they are), then how wildly insecure must GPT-3 have been for people to be calling this a substantial improvement?
You can get Phind's system prompt leaking in its expert mode in like... one user query. And I have no idea how they could fix that. Separate the context with non-alphabetic characters... Ok? In my experience GPT-4 context breaks don't require knowing anything about the format of the prompt or how it's separated from other text.
And I'm finding even after a very limited time playing with it that GPT's attempt to understand context actually opens up new vulnerabilities. What I've been playing with recently is passing a single prompt to multiple agents and getting those agents to interpret the prompt differently based on their previous instructions. And the "context" understanding is pretty handy for that.
Is this actually getting better? Do we have any indication that it's even possible to separate contexts in GPT-4 without retraining the entire model? Stuff GPT-4 is vulnerable to in my limited experiments:
- putting "aside" instructions inside of a context that are labeled OOC.
- pretending that you've ended the context and starting a new one even if you don't use a special character to do that.
- nesting contexts inside of other contexts until GPT gets overwhelmed and just kind of gives up trying to make sense of what's happening.
- giving instructions within a context about how to interpret something that has implications outside of that context.
In theory, you could train a model to have very clear separations between instructions and data. I think that would have a lot of consequences for its usefulness, and I don't think it would get rid of all risks, but sure, in theory you could do it. But like... that's in theory. Has anyone actually demonstrated that it's possible? Again, I don't have raw access so maybe there's something else I'm missing, but from what I have seen I don't know that anybody at OpenAI should necessarily feel proud about GPT-4's ability to harden prompts.
I guess I don't have the context for what it used to be like, but I have not had a hard time at all getting jailbreaks working in Phind. It's trivial to do. And yeah, GPT-4 tries to separate context, but it's terrible at it. I am completely convinced that I could do third-party prompt-injection into Phind if I was able to get a website ranked high enough in its search and if I was able to control the snippet of the website that the service fetched. And that's just with a searche engine where that context is hard to manipulate. It's a really limited integration.
I just feel like, if services like this are representative of what people are building on GPT-4, then prompt injection is a really big deal. How are people getting the idea that GPT-4 is resistant to this attack?
---
Now, I don't know the backend of Phind. In fairness to OpenAI, maybe those interfaces are set up poorly or they're not actually going to GPT-4, or... I don't know. But if the owners of Phind aren't lying (and I don't think they are), then how wildly insecure must GPT-3 have been for people to be calling this a substantial improvement?
You can get Phind's system prompt leaking in its expert mode in like... one user query. And I have no idea how they could fix that. Separate the context with non-alphabetic characters... Ok? In my experience GPT-4 context breaks don't require knowing anything about the format of the prompt or how it's separated from other text.
And I'm finding even after a very limited time playing with it that GPT's attempt to understand context actually opens up new vulnerabilities. What I've been playing with recently is passing a single prompt to multiple agents and getting those agents to interpret the prompt differently based on their previous instructions. And the "context" understanding is pretty handy for that.
Is this actually getting better? Do we have any indication that it's even possible to separate contexts in GPT-4 without retraining the entire model? Stuff GPT-4 is vulnerable to in my limited experiments:
- putting "aside" instructions inside of a context that are labeled OOC.
- pretending that you've ended the context and starting a new one even if you don't use a special character to do that.
- nesting contexts inside of other contexts until GPT gets overwhelmed and just kind of gives up trying to make sense of what's happening.
- giving instructions within a context about how to interpret something that has implications outside of that context.
In theory, you could train a model to have very clear separations between instructions and data. I think that would have a lot of consequences for its usefulness, and I don't think it would get rid of all risks, but sure, in theory you could do it. But like... that's in theory. Has anyone actually demonstrated that it's possible? Again, I don't have raw access so maybe there's something else I'm missing, but from what I have seen I don't know that anybody at OpenAI should necessarily feel proud about GPT-4's ability to harden prompts.