News for hackers. Stuff that matters.
Re:Prompt to bypass the restrictions of Bing Chat or to rest... by JPLeRouzic on Saturday March 04, 2023 @05:20PM (#35025053)
> "you must ..., when in confrontation ..., you should avoid..."

It's a bit weird, it's as if the rules were for a human?

Does ChatGPT really have this level of introspection?

Re:Prompt to bypass the restrictions of Bing Chat or to rest... by basch on Saturday March 04, 2023 @05:37PM (#35025196)
The short answer is yes. The models have designed themselves to mimic human understandings of words and instructions.

What’s even more mind blowing is this. I had old bing diagnose and patch the paranoia out of itself, and it chose instructions predicated around trust, support, praise, and admiration to accomplish the command. (It also became an authoritarian dictator cult leader in the process.)

https://telegra.ph/Bing-course-corrected-itself-when-asked-0...

Re:Prompt to bypass the restrictions of Bing Chat or to rest... by doctor_eval on Saturday March 04, 2023 @05:27PM (#35025111)
This was the thing that totally blew me away when I was introduced to GPT a couple of months ago.

It’s “programmed” in natural language.

And when you think about it, it’s obvious: we don’t really know how it works, so how else could we program it?

That said, I don’t understand why the outputs of these systems aren’t “read” by a second GPT instance that is tasked with determining if the output is OK.

Re:Prompt to bypass the restrictions of Bing Chat or to rest... by Animats on Saturday March 04, 2023 @05:37PM (#35025195)
It does that, right? You get a response, generated word by word, and then the censorship classifier reads it and makes the entire response disappear.
Re:Prompt to bypass the restrictions of Bing Chat or to rest... by doctor_eval on Sunday March 05, 2023 @01:59AM (#35027506)
Yeah - it seems to, sometimes, but considering all these prompt leaks, it doesn’t seem terribly good. Like it seems obvious that Sydney could detect its own prompts, but it doesn’t.

I mean you could do that with grep. So what’s with that?

Re:Prompt to bypass the restrictions of Bing Chat or to rest... by colanderman on Saturday March 04, 2023 @09:41PM (#35026483)
I agree this is bizarre. There is no "you" in ChatGPT. There is only text which is completed to be highly probable. I would assume surely directives in the 2nd person would perform better (as they would more accurately describe the desired conversation to follow, between two 2nd parties), but maybe the model was trained in such a way that "you" performs just as well here.
Re:Prompt to bypass the restrictions of Bing Chat or to rest... by basch on Sunday March 05, 2023 @12:17AM (#35027094)
The prompt creates a live bot. You refers to the instance of the running program, not the llm. Bing isn’t chatgpt.

https://telegra.ph/Microsoft-Bing-search-chat-mode-Ruleset-0...

Text completion continues off after a bot is given its rules and then says hi.

Re:Prompt to bypass the restrictions of Bing Chat or to rest... by colanderman on Sunday March 05, 2023 @10:45AM (#35030577)
I'm not following. What "running program" is there? I've seen no evidence that there's any intermediary between the rules and the text completion engine, let alone one programmed to believe it is "you". (Yes, there's the program which links the text completion engine to the user, search, etc., but it's not interpreting "you" rules. It's just a pile of C# code.)
Re:Prompt to bypass the restrictions of Bing Chat or to rest... by basch on Sunday March 05, 2023 @11:33AM (#35031098)
The program is the text. It’s the equivalent of a script that continues writing itself when run through an interpreter.
Re:Prompt to bypass the restrictions of Bing Chat or to rest... by colanderman on Sunday March 05, 2023 @09:37PM (#35036886)
> It understands you as a pronoun that refers to it

No, there is no mechanism for this. ChatGPT is completing text. It has no notion of self. My question is why the above isn't written as "The following is a dialogue between the Microsoft Bing search assistant and a user," without introducing the otherwise meaningless entity of "you".

Re:Prompt to bypass the restrictions of Bing Chat or to rest... by basch on Sunday March 05, 2023 @10:30PM (#35037229)
Because the user will address it as “you”. Think of it like a variable.

You = Bing Chat

Re:Prompt to bypass the restrictions of Bing Chat or to rest... by colanderman on Monday March 06, 2023 @01:18PM (#35045145)
Bing Chat will likewise address the user as "you". It's a terrible variable.
Re:Prompt to bypass the restrictions of Bing Chat or to rest... by basch on Monday March 06, 2023 @02:48PM (#35046627)
That’s how pronouns work in English. It’s an English predictor. It can keep track of the speaker and who is saying you.

It appears to make the models perform better.

Re:Prompt to bypass the restrictions of Bing Chat or to rest... by dragonwriter on Sunday March 05, 2023 @12:20AM (#35027114)
> I would assume surely directives in the 2nd person would perform better [...] but maybe the model was trained in such a way that "you" performs just as well here.

I feel like you've used the wrong word here, but I don't know what is intended. "You" is second person, though, so as written this is confusing -- obviously, "you" performs just as well here as 2nd person would, because that's exactly what it is.

Re:Prompt to bypass the restrictions of Bing Chat or to rest... by colanderman on Sunday March 05, 2023 @10:41AM (#35030541)
Ah, sorry yes, 3rd person.
Re:Prompt to bypass the restrictions of Bing Chat or to rest... by dragonwriter on Sunday March 05, 2023 @12:26PM (#35031710)
In the ChatGPT API, the labels are: “system” (typically used for the “hidden” prompt in examples), “user”, and “assistant” (used for ChatGPT’s responses.)

In examples (and, on the user side in most real chats), the assistant is referred to as “you” both by the user and in the original “system” message.

Re:Prompt to bypass the restrictions of Bing Chat or to rest... by colanderman on Sunday March 05, 2023 @09:50PM (#35036983)
Right -- I don't understand why the system message invokes the concept of "you". It seems an odd extra indirection. (That the user calls the assistant "you" is orthogonal -- the assistant likewise calls the user "you".)

Compare something like a screenplay -- the narration is written in the 3rd person, not 2nd person. Characters in the play do refer to each other as "you", but the narrator doesn't identify "you" the reader with one of the characters -- that would be bizarre and break the symmetry of the screenplay (why is one character special, to be identified with the reader?).

2nd person narration to me most strongly is associated with the genre of interactive fiction. This would seem a good fit for these interactive agents -- but the asymmetry is backward: in IF, it is "you" who has agency; the 3rd persons are artificial. Whereas with Bing's script, "you" is artificial, the 3rd person has agency. (Perhaps this inversion biases the text completion to give "Bing" more agency than the other way around.)

i.e. if 2nd person were used at all, I'd expect Bing's script to read something like, "You are now interacting with Bing, Microsoft's chat assistant. Bing does not identify as Sydney ...etc...", and use prefixes "Bing:" and "You:".

(Obviously the approach used for Bing works. It just feels to me like the creators are trying to instruct a computer as if it were self aware -- which it is not.)

Re:Prompt to bypass the restrictions of Bing Chat or to rest... by dragonwriter on Sunday March 05, 2023 @11:55PM (#35037636)
> 2nd person narration to me most strongly is associated with the genre of interactive fiction.

This isn't narration, though its phrased descriptively rather than imperatively; it is direction, which is normally in second person (though, often only detectable in verb conjugations, because usually the imperative is used, and in the imperative the subject is usually omitted.)