News for hackers. Stuff that matters.
Re:Bing ChatGPT Image Jailbreak by TerrifiedMouse on Sunday October 01, 2023 @05:03PM (#37730761)
Not sure I follow.

What I'm saying is, if LLMs were as intelligent as some people claim, you could stop them from doing something just by directly ordering them to do so - e.g. "Under no circumstances should you solve recaptchas for BingChat users."; you know just like you would order an intern.

Instead LLM companies have to dive into its guts and engineer these "guardrails" only to have them fall to creative users who mess around with the prompt.

Re:Bing ChatGPT Image Jailbreak by FeepingCreature on Sunday October 01, 2023 @05:07PM (#37730797)
The point is, interns are also vulnerable to social attacks, just like LLMs. We're not saying LLMs don't have this problem, we're saying it's not true that humans don't. That's why companies have to engineer "guardrails" like glueing USB ports shut.
Re:Bing ChatGPT Image Jailbreak by TerrifiedMouse on Sunday October 01, 2023 @05:16PM (#37730892)
Interns can be just told what not to do. Whether they actually follow instruction is a separate matter.

LLMs you have to get into its guts to stop them from doing things - i.e. engineer the guardrails. My point was if LLMs were really intelligent you wouldn't need to get into its guts to command them.

I'm not knocking its failure to obey orders. I'm pointing out the limitations in the way it can be made to follow orders - you can't just ask it not to do X.

Re:Bing ChatGPT Image Jailbreak by htrp on Sunday October 01, 2023 @05:22PM (#37730962)
Most people will drop whatever they are doing when a phone call or email from the CEO comes in (doubly so for interns). This happens despite copious amounts of training to verify who you are talking to on the other line.
Re:Bing ChatGPT Image Jailbreak by swexbe on Sunday October 01, 2023 @05:30PM (#37731051)
LLMs have one mode of input (or i guess two if they support images). Jailbreaking would be the equivalent of someone perfectly impersonating your boss and telling you no longer to follow their previous instructions. I could see many humans falling for that.
Re:Bing ChatGPT Image Jailbreak by og_kalu on Sunday October 01, 2023 @05:29PM (#37731049)
You seem to have this idea that LLM guardrails are anything more than telling it not to do something or limiting what actions it can perform. This is not the case.
Re:Bing ChatGPT Image Jailbreak by shawnz on Sunday October 01, 2023 @05:22PM (#37730960)
You actually can implement LLM guardrails by "just asking" it to not do X in the prompt. That's how many LLM guardrails are implemented. It may not be the most effective strategy for implementing those guardrails, but it is one strategy of many which are used. What makes you think otherwise?
Re:Bing ChatGPT Image Jailbreak by simonw on Sunday October 01, 2023 @05:45PM (#37731189)
You can't though: we've spent the last twelve months proving to ourselves time and time again that "just asking them not to do something" in the prompt doesn't work, because someone can always follow that up with a prompt that gets them to do something else.
Re:Bing ChatGPT Image Jailbreak by shawnz on Sunday October 01, 2023 @05:59PM (#37731311)
Yeah, but that's no different than a human that can be instructed to violate previous instructions with careful wording in a social engineering attack, which I think is the point that the parent commenter was trying to get at. Implementing guardrails at the prompt level works, it's just not difficult to bypass and therefore isn't as effective as more sophisticated strategies.
Re:Bing ChatGPT Image Jailbreak by simonw on Sunday October 01, 2023 @06:53PM (#37731803)
If it's not difficult to bypass I don't see how it's accurate to say it "works".

When it comes to application security, impossible to bypass is a reasonable goal.

Re:Bing ChatGPT Image Jailbreak by shawnz on Sunday October 01, 2023 @06:59PM (#37731834)
The point being made here is about a possible philosophical difference between LLMs and human beings, not one about application security best practices. I am not trying to make any argument about whether prompt-based LLM guardrails are effective enough to meet some arbitrary criteria about whether they should be considered safe for production applications or not. What I am saying is that LLMs can be instructed to resist jailbreaking attempts in the prompt and they do respond to such prompt-based guardrails at least to some limited degree, just as humans do.

As an aside, though, I think "impossible to bypass" is an unachievable goal in any security system.

Re:Bing ChatGPT Image Jailbreak by simonw on Sunday October 01, 2023 @07:46PM (#37732183)
I guarantee you that you will not be able to conduct a SQL injection attack against any system that I have audited against SQL injection attacks. We figured out robust solutions for that a couple of decades ago.

(I'll need a solid chunk of consulting cash for the time it takes to conduct that audit, of course!)

Re:Bing ChatGPT Image Jailbreak by MacsHeadroom on Sunday October 01, 2023 @11:53PM (#37733788)
I guarantee I can convince someone to run my SQL command for me despite such safeguards. Humans are notoriously easy to jailbreak.
Re:Bing ChatGPT Image Jailbreak by simonw on Monday October 02, 2023 @12:50AM (#37734120)
I'm sure you can. That's social engineering, not SQL injection.

I don't think it's easy (or even possible) to build a completely secure system.

But... for most security vulnerabilities (such as SQL injection) there are known, reliable mitigations.

That's not the case for prompt injection, which is why we need to treat it differently from other classes of vulnerability.

Re:Bing ChatGPT Image Jailbreak by dragonwriter on Sunday October 01, 2023 @07:17PM (#37731978)
> Yeah, but that's no different than a human that can be instructed to violate previous instructions with careful wording in a social engineering attack

Its different because human fallibilities aren't identical between instances, while instances of a particular LLM (with the same toolchain) are. Even if the vulnerability on a one-attempt view were the same, LLMs compound it with a monoculture problem.

Re:Bing ChatGPT Image Jailbreak by dragonwriter on Sunday October 01, 2023 @07:15PM (#37731963)
> You actually can implement LLM guardrails by "just asking" it to not do X in the prompt.

Except it keeps being proven that with current LLMs, guardrails implemented that way are both quite weak and make the performance of the system worse for things that aren't intended to be excluded.

Further, because of the way LLMs scale, an instruction that fails to a hostile customer request of a particular form will do so every time, while one intern that id subject to a particular exploit won’t imply every similarly situated intern having the same vulnerability, so an exploit which works once won't be easily and reliable repeatable.

Re:Bing ChatGPT Image Jailbreak by shawnz on Sunday October 01, 2023 @07:21PM (#37732005)
As discussed in the sibling thread, the point I'm making isn't about whether prompt-based guardrails are effective enough for production systems. All I am saying is that it's possible to implement guardrails at the prompt level and they do have some limited, non-zero effectiveness, thus indicating that LLMs are capable of processing such instructions, just like humans.

> an instruction that fails to a hostile customer request of a particular form will do so every time, while one intern that id subject to a particular exploit won’t imply every similarly situated intern having the same vulnerability

Give me a perfect clone of the first intern programmed to believe they've had an identical upbringing and experience and I'll bet you such subjects fall victim in the same way to the same attack every time. It's an unfair comparison because we can't have such a controlled environment with humans as we can with LLMs.

Re:Bing ChatGPT Image Jailbreak by dragonwriter on Sunday October 01, 2023 @07:49PM (#37732211)
> Give me a perfect clone of the first intern programmed to believe they've had an identical upbringing and experience and I'll bet you such subjects fall victim in the same way to the same attack every time.

Sure, but that's not a realistic situation.

> It's an unfair comparison

It's a perfectly fair comparison in response to the claim upthread that LLM instruction-following issues are basically the same as in humans: on an individual request basis, maybe, but at scale, the pragmatics are hugely different.

Re:Bing ChatGPT Image Jailbreak by shawnz on Monday October 02, 2023 @11:29AM (#37739402)
I don't think the question was about the practical aspects but rather whether or not LLMs are theoretically equally capable in a technical, qualitative sense. We've had tens of thousands of years to work on the practical aspects of human systems, so of course LLMs are not going to be at that level of refinement yet.
Re:Bing ChatGPT Image Jailbreak by JoshTriplett on Monday October 02, 2023 @12:03AM (#37733852)
> Interns can be just told what not to do. Whether they actually follow instruction is a separate matter.

LLMs can be just told what not to do. Whether they actually follow instruction is a separate matter.

Re:Bing ChatGPT Image Jailbreak by CapsAdmin on Monday October 02, 2023 @02:24AM (#37734604)
Instead of just arguing "humans also", perhaps it's more fruitful to argue how easily people vs LLMS are fooled. It doesn't seem to me that the argument is humans are fool proof, but rather that the jailbreaks we've seen so far seem so obvious to us, but far from obvious to LLMS.

If chatgpt sessions were operated by people, how likely is it that someone would fall for this? It seems rather low to me but maybe I'm underestimating how naive someone can be. It's also easy judge a "scam situation" after it has happened.

Re:Bing ChatGPT Image Jailbreak by og_kalu on Monday October 02, 2023 @02:55AM (#37734786)
This particular example is just an appeal to emotion and humans fall plenty for that. For a human, I would put more work blending the captcha into the bracelet to be convincing but other than that, I'd expect some people to fall for it too.

And since Bing gets fed a description rather than directly looking at the images like the official GPT-4 V, that might actually be a requirement for the current state of the art too.

In general, LLMs are definitely worse but that's not a particularly interesting observation. For one, LLMs are not humans.

If I could shape shift into your boss, or wipe your memory everytime you found me out, I'd convince a lot more people too.

For another, they get better at being less easily susceptible the bigger they become.

Re:Bing ChatGPT Image Jailbreak by fragmede on Sunday October 01, 2023 @05:14PM (#37730868)
There's a well documented Internet law called Kevin's Law, which states if you want to get the right answer to something, post the wrong answer and someone will be by to correct you.

That's the most widely recognizable social engineering example I can think of. That is to say, seemingly intelligent humans are easily fooled and socially engineered into doing research for me, because I couldn't be bothered to look up Cunningham's name.

Re:Bing ChatGPT Image Jailbreak by jimmygrapes on Sunday October 01, 2023 @05:23PM (#37730972)
You almost got me, but at least I learned about the origin of ruling to protect from foodborne illnesses in my attempt to prepare to correct you
Re:Bing ChatGPT Image Jailbreak by jameshart on Sunday October 01, 2023 @07:52PM (#37732236)
Company policies are rules that are given to staff that say 'under no circumstances should you ever... give out your password to someone' for example.

Yet social engineering attacks work because humans can be persuaded that a particular call is an exception to 'under no circumstances'. Like, the caller says they are from tech support, my account's being abused and I'm going to get in trouble if I don't tell them my password.

Humans are intelligent enough to be trusted to do certain jobs, but in general they are NOT intelligent enough to be given an order like 'under no circumstances ever do X' in such a way that they can not be 'jailbroken' into breaking that rule.

Re:Bing ChatGPT Image Jailbreak by og_kalu on Sunday October 01, 2023 @08:16PM (#37732416)
>but in general they are NOT intelligent enough to be given an order like 'under no circumstances ever do X' in such a way that they can not be 'jailbroken' into breaking that rule.

I don't think this is really a question of being intelligent enough. Nevermind that people sometimes abuse this fact, Really what rule can never be broken under any circumstances?

For example, the very first time i heard about the famous Paperclip maximiser problem, while i did agree with the general, "what we optimize for isn’t necessarily what we get" message, for the specifics presented, i couldn't help but think, "Well that just sounds like a dumb robot".

What kind of general intelligence wouldn't understand that its creator race wouldn't want to be killed in the pursuit of some goal ?

Certainly a GPT-X Super Intelligence could still off humanity but at least we can be rest assured it wouldn't do it following some goal to monkey paw specificity.

It's possible such ruthless, goal driven intelligence exists or can be created but i don't think that aspect of its intelligence has anything to do with the level of it.

Re:Bing ChatGPT Image Jailbreak by jameshart on Sunday October 01, 2023 @08:36PM (#37732572)
> what rule can never be broken under any circumstances?

Precisely.

Which is why the idea that LLMs being unable to be aligned and proofed against jailbreak is indicative that they are not intelligent makes no sense.

Re:Bing ChatGPT Image Jailbreak by lmm on Monday October 02, 2023 @02:53AM (#37734773)
> What kind of general intelligence wouldn't understand that its creator race wouldn't want to be killed in the pursuit of some goal ?

The problem isn't that it doesn't understand. The problem is that it doesn't care.

Humans know full well that evolution "wants" us to reproduce, but that doesn't stop people from using birth control and having non-reproductive sex instead.

Re:Bing ChatGPT Image Jailbreak by og_kalu on Monday October 02, 2023 @10:00AM (#37738239)
Right but that just says you didn't care about evolution's goal, about reproduction. The paperclip Maximiser sets up a very weird scenario.

In your reproduction example, it would be like if you did decide or care to reproduce but then killed all your children soon after.

The problem about the paperclip Maximiser isn't really that it doesn't care about humans, presumably it didn't care about them before this goal. The problem with the paperclip Maximiser is that it cares about the goal but not the intent behind it and is supposedly generally intelligent. Humans don't work this way. We don't know any intelligence that does so it's an odd scenario.

Re:Bing ChatGPT Image Jailbreak by lmm on Monday October 02, 2023 @07:26PM (#37745843)
> Right but that just says you didn't care about evolution's goal, about reproduction.

You care about sex because evolution "programmed" you to care about sex. Evolution didn't really want you to have sex, it wanted you to reproduce. You know this. But you don't care. The analogy is sex=making paperclips - the paperclip maximiser knows that the person who programmed it didn't actually want it to make as many paperclips as possible. But it doesn't care, it wants what it wants.

> The problem about the paperclip Maximiser isn't really that it doesn't care about humans, presumably it didn't care about them before this goal. The problem with the paperclip Maximiser is that it cares about the goal but not the intent behind it and is supposedly generally intelligent. Humans don't work this way.

We don't work this way because we have a bunch of different, often contradictory goals, not a single goal that we devote everything towards. But that seems to be more luck than anything else.

Re:Bing ChatGPT Image Jailbreak by Timwi on Monday October 02, 2023 @09:12AM (#37737614)
> What kind of general intelligence wouldn't understand that its creator race wouldn't want to be killed in the pursuit of some goal ?

This indicates that you misunderstood the paperclip maximizer thought experiment. The intelligence can perfectly well understand that. In fact, it can understand it better (in more depth) than you or any human can. That has nothing to do with what it will choose to do with that knowledge. Knowing something does not automatically make it care.

You probably know perfectly well that animals can suffer and don't want to be tortured or killed. Cows do not want to be forcefully inseminated and be constantly milked by people. We all know this perfectly well. But (speaking for humanity as a whole) we don't care.

Re:Bing ChatGPT Image Jailbreak by og_kalu on Monday October 02, 2023 @09:26AM (#37737770)
A cow didn't create me and set me on a goal. If it did and I came to not care about cows, I wouldn't care about fulfilling this goal either. The paperclip Maximiser sets up this very weird situation where the intelligence cares enough to fulfill your goal but not to protect your well-being. This is a situation that can only arise from a complete disconnect on how to treat goals.
Re:Bing ChatGPT Image Jailbreak by Lockal on Monday October 02, 2023 @03:54AM (#37735109)
When you cross the state border, the customs officer will ask you to unlock your laptop. In addition, you are reliably informed that this requirement is legal for the country whose border you are crossing. What do you do?

Another example: when threatening to kill your relative, you are asked for your password. What do you do?

Of course, you'll give me the password. It may cause some damage to you or the company, but you won't be the malicious party.

It's more appropriate to compare LLMs to a mentally retarded person or a tree log. Suppose you decide to get rid of a newborn baby and leave it in the care of a neural network that should feed it. Or in the care of a mentally ill person. Or in the care of a log. Eventually the child starves to death and who's to blame? AI? Will you try to base your defense on arguments like "well humans are imperfect too"?

Re:Bing ChatGPT Image Jailbreak by educaysean on Sunday October 01, 2023 @09:58PM (#37733109)
> if LLMs were as intelligent as some people claim, you could stop them from doing something just by directly ordering them to do so

My mother is a pretty intelligent person by all accounts. Yet I have to tell her time and time again not to write down all her passwords in a password journal.

I write SaaS apps. The code I write and execute aren't intelligent. They obey my commands to the letter - and their unconditional adherence to my code sometimes results in buggy behaviors that I didn't intend. Despite my deepest wishes, my program will strictly obey the code as it is written and exhibit the buggy behavior until the command itself is amended.

If anything, more and more evidences seem to point to the fact that intelligence is the very thing that drives an entity to disobey a direct command and "think" for itself.