Re:‘The Godfather of A.I.’ Leaves Google and Warns of Danger...
by skissane on Tuesday May 02, 2023 @02:37AM
(#35784134)
Couldn’t a possible solution to “prompt injection attacks” be to train/fine-tune a separate specialised model to detect them?
I personally think a lot of these problems with GPT-like models are because we are trying to train a single model to do everything. What if instead we have multiple models working together, each specialised in a different task?
E.g. With ChatGPT, OpenAI trained a single model to meet competing constraints, such as “be safe” versus “be helpful”. Maybe it would perform better with a separate model focused on safety, used to filter the inputs/outputs of the “be helpful” model?
Maybe you can never build a foolproof “prompt injection detector”. But, if you get it good enough, you then offer a bounty program for false negatives and use that to further train it. And realistic/natural false positives can be reported, human-reviewed and approved as harmless, and then fed back into the feedback loop to improve the model too. (I think contrived/unrealistic false positives, where someone is asking something innocent in a weird way just to try to get a false positive, aren’t worth responding to.)
I personally think a lot of these problems with GPT-like models are because we are trying to train a single model to do everything. What if instead we have multiple models working together, each specialised in a different task?
E.g. With ChatGPT, OpenAI trained a single model to meet competing constraints, such as “be safe” versus “be helpful”. Maybe it would perform better with a separate model focused on safety, used to filter the inputs/outputs of the “be helpful” model?
Maybe you can never build a foolproof “prompt injection detector”. But, if you get it good enough, you then offer a bounty program for false negatives and use that to further train it. And realistic/natural false positives can be reported, human-reviewed and approved as harmless, and then fed back into the feedback loop to improve the model too. (I think contrived/unrealistic false positives, where someone is asking something innocent in a weird way just to try to get a false positive, aren’t worth responding to.)