News for hackers. Stuff that matters.
Re:Delimiters won’t save you from prompt injection by squeaky-clean on Saturday May 13, 2023 @03:19AM (#35926341)
Have you managed to break recent versions of Bing Chat? I've tried some of the notable injection prompts and my own variants. Sometimes I can get it to begin with the jailbreak, but after 10-20 words it will erase the previous text and say something along the lines of "I cannot talk about that"
Re:Delimiters won’t save you from prompt injection by codeflo on Saturday May 13, 2023 @03:33AM (#35926409)
My reading of these events is that the jailbreak is successful, but some other, slower system detects it after the fact and deletes the message. Given the delay, I've seen speculation that there's a manual step involved.
Re:Delimiters won’t save you from prompt injection by simonw on Saturday May 13, 2023 @09:40AM (#35928268)
Jailbreaking and prompt injection are related but they aren't quite the same problem.

That thing where Bing redacts text after the point isn't going to protect your application against problems like those I describe in https://simonwillison.net/2023/Apr/14/worst-that-can-happen/

Re:Delimiters won’t save you from prompt injection by squeaky-clean on Monday May 15, 2023 @08:13PM (#35956289)
Great point