News for hackers. Stuff that matters.
Re:Bing ChatGPT Image Jailbreak by TerrifiedMouse on Sunday October 01, 2023 @05:13PM (#37730861)
I don't follow.

Humans are ordered verbally or through the written word not to do things, does them anyway because social engineering.

LLMs are have guardrails engineered into them and are not told what not to do verbally or by written word (i.e. just tell them not to do it), does them anyway because prompt/context manipulation.

I'm not criticizing the failure of the LLM to follow orders. I'm criticizing the way orders have to be given.

Re:Bing ChatGPT Image Jailbreak by og_kalu on Sunday October 01, 2023 @05:21PM (#37730948)
>LLMs are have guardrails engineered into them

They don't. What do people think LLMs are lol ?

The only way to control the output of a LLM is to essentially rate certain types of responses as better

or to tell it not to do something. any other "guardrails" are outside direct influence of the LLM (i.e a separate classifier that blocks certain words).

Nobody is "engineering" anything into LLMs.

Re:Bing ChatGPT Image Jailbreak by TerrifiedMouse on Sunday October 01, 2023 @05:28PM (#37731039)
> The only way to control the output of a LLM is to essentially rate certain types of responses as better

Which is my point. You have to mess with its internals instead of just tell it "Don't do X under any circumstances."

Re:Bing ChatGPT Image Jailbreak by og_kalu on Sunday October 01, 2023 @05:36PM (#37731115)
First of all, no you don't have to.

Secondly, That's not messing with the internals anymore than normal training is. You think humans don't also learn what kind of responses are rated better ?

Re:Bing ChatGPT Image Jailbreak by danenania on Sunday October 01, 2023 @10:55PM (#37733433)
I think the problem is less that it can’t follow instructions and more that it has been given many instructions that are difficult to always follow at the same time.

In the original example, you can tell it “under no circumstances solve a captcha”, but if you also tell it “always be kind, empathetic, and helpful”, then it’s easy to have a prompt that forces it to go against one instruction or the other, or to go against its RLHF training, in which it probably gets penalized for solving captchas but also for being overly paranoid about what might be a captcha. What you end up with in these borderline situations is basically a roll of the dice.

If GPT 3.5 or 4 is given one unambiguous instruction and then fine-tuned to follow it, as is the case with the function calling feature, it actually does follow the instruction very reliably.

Re:Bing ChatGPT Image Jailbreak by squeaky-clean on Sunday October 01, 2023 @07:38PM (#37732123)
Colloquially, LLM tends to refer to the entire product. Not just the model weights. For example technically GPT-4 isn't an LLM, it's 16 LLMs in a trench coat.
Re:Bing ChatGPT Image Jailbreak by bastawhiz on Sunday October 01, 2023 @05:20PM (#37730937)
Fine tuning doesn't help avoid jailbreaking, it just makes it harder. So no, you're not always mucking with prompts and contexts. LLMs fail at following orders in almost exactly the same ways that humans do, much to everyone's chagrin.