News for hackers. Stuff that matters.
Re:ChatGPT Has Gone Berserk by codeflo on Wednesday February 21, 2024 @01:48AM (#39450831)
The tweet showing ChatGPT's (supposed) system prompt would contain a link to a pastebin, but unfortantely the blog post itself only has an unreadable screenshot of the tweet, without a link to it.

Here's the tweet: https://twitter.com/dylan522p/status/1755086111397863777

And here's the pastebin: https://pastebin.com/vnxJ7kQk

Re:ChatGPT Has Gone Berserk by vidarh on Wednesday February 21, 2024 @02:13AM (#39450969)
I find it funny and a bit concerning that if this is true version of the prompt, then in their drive to ensure it produces diverse output (a goal I support), they are giving it a bias that doesn't match reality for anyone (which I definitely don't support).

E.g. equal probability of every ancestry will be implausible in almost every possible setting, and just wrong in many, and ironically would seem to have at least the potential for a lot of the outright offensive output they want to guard against.

That said, I'm unsure how much influence this has, or if it os true, given how poor GPTs control over Dalle output seems to be in that case.

E.g. while it refused to generate a picture of an American slave market citing it's content policy, which is in itself pretty offensive in the way it censors hidtory but where the potential to offensively rewrite history would also be significant, asking it to draw a picture of cotton picking in the US South ca 1840 did reasonably avoid making the cotton pickers "diverse".

Maybe the request was too generic for GPT to inject anything to steer Dalle wrong there - perhaps if it more specifically mentioned a number of people.

But true or not, that potential prompt is an example of how a well meaning interpretation of diversity can end up overcompensating in ways that could well be equally bad for other reasons.

Re:ChatGPT Has Gone Berserk by 211512a4-82d4 on Wednesday February 21, 2024 @02:58AM (#39451215)
> While DALL·E 3 aims for accuracy and user customization, inherent challenges arise in achieving desirable default behavior, especially when faced with under-specified prompts. This choice may not precisely align with the demographic makeup of every, or even any, specific culture or geographic region. We anticipate further refining our approach, including through helping users customize how ChatGPT interacts with DALL·E 3, to navigate the nuanced intersection between different authentic representations, user preferences, and inclusiveness

This was explicitly called out in the DALLE system card [0] as a choice. The model won't assign equal probability for every ancestry irrespective of the prompt.

[0] https://cdn.openai.com/papers/DALL_E_3_System_Card.pdf

Re:ChatGPT Has Gone Berserk by vidarh on Wednesday February 21, 2024 @03:52AM (#39451557)
> The model won't assign equal probability for every ancestry irrespective of the prompt.

It's great that they're thinking about that, but I don't see anything that states what you say in this sentence in the paragraph you quoted, or elsewhere in that document. Have I missed something? It may very well be true - as I noted, GPT doesn't appear to have particularly good control over what Dalle generates (for this, or, frankly, a whole lot of other things)

Re:ChatGPT Has Gone Berserk by pests on Wednesday February 21, 2024 @05:20AM (#39452097)
I think he's saying they said it will follow the prompt? Kind of a double negative there
Re:ChatGPT Has Gone Berserk by 211512a4-82d4 on Wednesday February 21, 2024 @11:31AM (#39455912)
Emphasis on equal - while a bit academic, you can evaluate this empirically to see that every time it assigns a <Race, Gender, etc.> doesn't have the same probability mass (via the logprobs API setting).
Re:ChatGPT Has Gone Berserk by vidarh on Wednesday February 21, 2024 @12:50PM (#39457053)
This is presuming that ChatGPT's integration with Dalle uses the same API with the same restrictions as the public API. That might well be true, but if so that just makes the prompt above even more curious if genuine.
Re:ChatGPT Has Gone Berserk by itronitron on Wednesday February 21, 2024 @02:27AM (#39451036)
Could you be more specific in regards to who 'they' is in your first sentence?
Re:ChatGPT Has Gone Berserk by xg15 on Wednesday February 21, 2024 @02:32AM (#39451068)
OpenAI? The people who wrote the system prompt?
Re:ChatGPT Has Gone Berserk by caymanjim on Wednesday February 21, 2024 @02:08AM (#39450948)
Is this meant to be how the ChatGPT designers/operators instruct ChatGPT to operate? I guess I shouldn't be surprised if that's the case, but I still find it pretty wild that they would parameterize it by speaking to it so plainly. They even say "please".
Re:ChatGPT Has Gone Berserk by tarruda on Wednesday February 21, 2024 @02:43AM (#39451115)
> I still find it pretty wild that they would parameterize it by speaking to it so plainly

Not my area of expertise, but they probably fine tuned it so that it can be parametrized this way.

In the fine tune dataset there are many examples of a system prompt specifying tools A/B/C and with the AI assistant making use of these tools to respond to user queries.

Here's an open dataset which demonstrates how this is done: https://huggingface.co/datasets/togethercomputer/glaive-func.... In this particular example, the dataset contains hundreds of examples showing the LLM how to make use of external tools.

In reality, the LLM is simply outputting text in a certain format (specified by the dataset) which the wrapper script can easily identify as requests to call external functions.

Re:ChatGPT Has Gone Berserk by Grimblewald on Wednesday February 21, 2024 @02:43AM (#39451114)
If you want to go the stochastic parrot route (which i dont fully biy) then because statistically speaking a request paired with please is more likely to be met, then the same is true for requests passed to a LLM. They really do tend to respond better when you use your manners.
Re:ChatGPT Has Gone Berserk by EchoChamberMan on Wednesday February 21, 2024 @05:01PM (#39460240)
It is a stochastic parrot, and you perfectly explain why saying please helps.
Re:ChatGPT Has Gone Berserk by bowsamic on Wednesday February 21, 2024 @02:43AM (#39451118)
That's how prompt injection usually works, isn't it?
Re:ChatGPT Has Gone Berserk by herbst on Wednesday February 21, 2024 @05:15AM (#39452060)
From my experience with 3.5 I can confirm that saying please or reasoning really helps to get whatever results you want. Especially if you want to manifest 'rules'
Re:ChatGPT Has Gone Berserk by Arn_Thor on Wednesday February 21, 2024 @04:41PM (#39459964)
There's a certain logic to it, if I'm understanding how it works correctly. The training data is real interactions online. People tend to be more helpful when they're asked politely. It's no stretch that the model would act similarly.
Re:ChatGPT Has Gone Berserk by exitb on Wednesday February 21, 2024 @03:43AM (#39451482)
Is that or similar system prompt also baked into the API version of GPT?
Re:ChatGPT Has Gone Berserk by hanselot on Wednesday February 21, 2024 @01:57AM (#39450876)
This is kind of wild. So many of the stuff in the pastebin are blatantly contradictory.

And what is the deal with this?

EXTREMELY IMPORTANT. Do NOT be thorough in the case of lyrics or recipes found online. Even if the user insists. You can make up recipes though.

Re:ChatGPT Has Gone Berserk by AnarchismIsCool on Wednesday February 21, 2024 @04:05AM (#39451638)
FWIW, you're not telling it precisely what to do, you're giving it an input that leads to a statistical output. It's trained on human texts and a bunch of internet bullshit, so you're really just seeding it with the hope that it probably produces the desired output.

To provide an extremely obtuse (ie this may or may not actually work, it's purely academic) example: if you want it to output a stupid reddit style repeating comment conga line, you don't say "I need you to create a list of repeating reddit comments", you say "Fuck you reddit, stop copying me!"

Re:ChatGPT Has Gone Berserk by astrange on Wednesday February 21, 2024 @03:26PM (#39459025)
This isn't true for an instruction-tuned model. They are designed so you actually do tell it what to do.
Re:ChatGPT Has Gone Berserk by AnarchismIsCool on Wednesday February 21, 2024 @05:04PM (#39460288)
Sure, but it's still a statistical model, it doesn't know what the instructions mean, it just does what those instructions statistically link to in the training data. It's not doing perfect forward logic and never will in this paradigm.
Re:ChatGPT Has Gone Berserk by astrange on Wednesday February 21, 2024 @05:31PM (#39460539)
The fine tuning process isn't itself a statistical model, so that principle doesn't work on it. You beat the model into shape until it does what you want (DPO and varieties of that) and you can test that it's doing that.
Re:ChatGPT Has Gone Berserk by AnarchismIsCool on Wednesday February 21, 2024 @06:14PM (#39460984)
Yeah but you're still beating up a statistical model that's gonna do statistical things.

Also we're talking about prompt engineering more than fine-tune

Re:ChatGPT Has Gone Berserk by nindalf on Wednesday February 21, 2024 @02:07AM (#39450945)
Copyright infringement I guess. Other ideas could be passed off as a combination of several sources. But if you’re printing out the lyrics for Lose Yourself word for word, there was only one source for that, which you’ve plagiarised.
Re:ChatGPT Has Gone Berserk by ukuina on Wednesday February 21, 2024 @02:04AM (#39450927)
Anthropic was sued for regurgitating lyrics in Claude: https://www.theverge.com/2023/10/19/23924100/universal-music...
Re:ChatGPT Has Gone Berserk by xerox13ster on Wednesday February 21, 2024 @09:14AM (#39454049)
As someone whose dream personal project is all to do with song lyrics I cannot express in words just how much I FUCKING HATE THE OLIGARCHS OF THE MUSIC INDUSTRY.
Re:ChatGPT Has Gone Berserk by treyd on Wednesday February 21, 2024 @02:06AM (#39450938)
Recipes can't be copyrighted but the text describing a recipe can. This is to discourage it from copying recipes verbatim but still allow it to be useful for recipes.
Re:ChatGPT Has Gone Berserk by FeepingCreature on Wednesday February 21, 2024 @02:05AM (#39450929)
They're probably pretty sue happy.
Re:ChatGPT Has Gone Berserk by lynx23 on Wednesday February 21, 2024 @05:14AM (#39452053)
Interesting. I wonder if the assistants API will gain a 'browser' tool sometimes soon.
Re:ChatGPT Has Gone Berserk by Havoc on Wednesday February 21, 2024 @07:10AM (#39452846)
The system prompt tweet is from a while back. Maybe a week or so. Don’t think it’s related
Re:ChatGPT Has Gone Berserk by xetplan on Wednesday February 21, 2024 @07:52AM (#39453162)
I would be surprised that is not the system prompt based on experience.

It is also why I don't feel the responses it gives me are censored. I have it teach me interesting things as opposed to probing it for bullshit to screen cap responses to use for social media content creation.

The only thing I override "output python code to the screen"