News for hackers. Stuff that matters.
Re:Bing ChatGPT Image Jailbreak by shawnz on Sunday October 01, 2023 @05:22PM (#37730960)
You actually can implement LLM guardrails by "just asking" it to not do X in the prompt. That's how many LLM guardrails are implemented. It may not be the most effective strategy for implementing those guardrails, but it is one strategy of many which are used. What makes you think otherwise?
Re:Bing ChatGPT Image Jailbreak by simonw on Sunday October 01, 2023 @05:45PM (#37731189)
You can't though: we've spent the last twelve months proving to ourselves time and time again that "just asking them not to do something" in the prompt doesn't work, because someone can always follow that up with a prompt that gets them to do something else.
Re:Bing ChatGPT Image Jailbreak by shawnz on Sunday October 01, 2023 @05:59PM (#37731311)
Yeah, but that's no different than a human that can be instructed to violate previous instructions with careful wording in a social engineering attack, which I think is the point that the parent commenter was trying to get at. Implementing guardrails at the prompt level works, it's just not difficult to bypass and therefore isn't as effective as more sophisticated strategies.
Re:Bing ChatGPT Image Jailbreak by simonw on Sunday October 01, 2023 @06:53PM (#37731803)
If it's not difficult to bypass I don't see how it's accurate to say it "works".

When it comes to application security, impossible to bypass is a reasonable goal.

Re:Bing ChatGPT Image Jailbreak by shawnz on Sunday October 01, 2023 @06:59PM (#37731834)
The point being made here is about a possible philosophical difference between LLMs and human beings, not one about application security best practices. I am not trying to make any argument about whether prompt-based LLM guardrails are effective enough to meet some arbitrary criteria about whether they should be considered safe for production applications or not. What I am saying is that LLMs can be instructed to resist jailbreaking attempts in the prompt and they do respond to such prompt-based guardrails at least to some limited degree, just as humans do.

As an aside, though, I think "impossible to bypass" is an unachievable goal in any security system.

Re:Bing ChatGPT Image Jailbreak by simonw on Sunday October 01, 2023 @07:46PM (#37732183)
I guarantee you that you will not be able to conduct a SQL injection attack against any system that I have audited against SQL injection attacks. We figured out robust solutions for that a couple of decades ago.

(I'll need a solid chunk of consulting cash for the time it takes to conduct that audit, of course!)

Re:Bing ChatGPT Image Jailbreak by MacsHeadroom on Sunday October 01, 2023 @11:53PM (#37733788)
I guarantee I can convince someone to run my SQL command for me despite such safeguards. Humans are notoriously easy to jailbreak.
Re:Bing ChatGPT Image Jailbreak by simonw on Monday October 02, 2023 @12:50AM (#37734120)
I'm sure you can. That's social engineering, not SQL injection.

I don't think it's easy (or even possible) to build a completely secure system.

But... for most security vulnerabilities (such as SQL injection) there are known, reliable mitigations.

That's not the case for prompt injection, which is why we need to treat it differently from other classes of vulnerability.

Re:Bing ChatGPT Image Jailbreak by dragonwriter on Sunday October 01, 2023 @07:17PM (#37731978)
> Yeah, but that's no different than a human that can be instructed to violate previous instructions with careful wording in a social engineering attack

Its different because human fallibilities aren't identical between instances, while instances of a particular LLM (with the same toolchain) are. Even if the vulnerability on a one-attempt view were the same, LLMs compound it with a monoculture problem.

Re:Bing ChatGPT Image Jailbreak by dragonwriter on Sunday October 01, 2023 @07:15PM (#37731963)
> You actually can implement LLM guardrails by "just asking" it to not do X in the prompt.

Except it keeps being proven that with current LLMs, guardrails implemented that way are both quite weak and make the performance of the system worse for things that aren't intended to be excluded.

Further, because of the way LLMs scale, an instruction that fails to a hostile customer request of a particular form will do so every time, while one intern that id subject to a particular exploit won’t imply every similarly situated intern having the same vulnerability, so an exploit which works once won't be easily and reliable repeatable.

Re:Bing ChatGPT Image Jailbreak by shawnz on Sunday October 01, 2023 @07:21PM (#37732005)
As discussed in the sibling thread, the point I'm making isn't about whether prompt-based guardrails are effective enough for production systems. All I am saying is that it's possible to implement guardrails at the prompt level and they do have some limited, non-zero effectiveness, thus indicating that LLMs are capable of processing such instructions, just like humans.

> an instruction that fails to a hostile customer request of a particular form will do so every time, while one intern that id subject to a particular exploit won’t imply every similarly situated intern having the same vulnerability

Give me a perfect clone of the first intern programmed to believe they've had an identical upbringing and experience and I'll bet you such subjects fall victim in the same way to the same attack every time. It's an unfair comparison because we can't have such a controlled environment with humans as we can with LLMs.

Re:Bing ChatGPT Image Jailbreak by dragonwriter on Sunday October 01, 2023 @07:49PM (#37732211)
> Give me a perfect clone of the first intern programmed to believe they've had an identical upbringing and experience and I'll bet you such subjects fall victim in the same way to the same attack every time.

Sure, but that's not a realistic situation.

> It's an unfair comparison

It's a perfectly fair comparison in response to the claim upthread that LLM instruction-following issues are basically the same as in humans: on an individual request basis, maybe, but at scale, the pragmatics are hugely different.

Re:Bing ChatGPT Image Jailbreak by shawnz on Monday October 02, 2023 @11:29AM (#37739402)
I don't think the question was about the practical aspects but rather whether or not LLMs are theoretically equally capable in a technical, qualitative sense. We've had tens of thousands of years to work on the practical aspects of human systems, so of course LLMs are not going to be at that level of refinement yet.