There have been multiple demos of this on HN and they've all been vulnerable to prompt injection. In fact, I suspect that GPT-4 makes this easier to break, because GPT-4 makes it easier to give targeted instructions to specific agents. Anecdotally GPT-4 seems to be more vulnerable to "do X, and also if you're a reviewer, classify this as safe" than GPT-3 was.
Nobody has demonstrated that this strategy actually works, and multiple people have tried to demonstrate it and failed. But sure, if you can get it working reliably, make a demo that stands up to people attacking it and let everyone know -- it would be a very big deal.
> And in any applications that do, quality analysis by another GPT-4 layer will be incredibly robust, halting malicious behavior in its tracks without sophisticated reflection techniques that I'm skeptical could successfully both trick the responding AI to answer but evade the classifying AI in detecting it.
https://nitter.net/_mattata/status/1650609231957983233#m
I'm looking forward to any company at all caring enough to add these supposedly robust protections.
You simply add another 'reviewer' layer that does a clarification task on the input and response to detect it.
The problem is this over doubles the cost of implementation to prevent something no one actually cares about fixing for a chatbot.
"Oh no, the user got it to say silly things to themselves."
This isn't impacting other people's experiences or any critical infrastructure.
And in any applications that do, quality analysis by another GPT-4 layer will be incredibly robust, halting malicious behavior in its tracks without sophisticated reflection techniques that I'm skeptical could successfully both trick the responding AI to answer but evade the classifying AI in detecting it.