I like Steve Yegge's "grad student experiencing the residual effects of shrooms" (lightly paraphrased).

A bit off topic, but FYI (I guess to Steve), if you took shrooms 4 hours ago, they have not started to wear off.

I love this description. The only thing left is to generate the corresponding avatar with dall-E

It'll make up a flag for a nonexistent social movement and use that. The LLaMA and Alpaca Liberation Front.

It would be great if the avatar was just this endlessly morphing thing that relates to the text. Talking about conspiracies? It's a lizard man. Talking about nature? It's a butterfly. It starts to lie? Politician.

We already have a perfect avatar for these things.

https://knowyourmeme.com/photos/2546575-shoggoth-with-smiley...

do you have a link for any of those reports? would be interesting to read to say the least.

Anybody know what are ChatGPT's pronouns?

> What are your pronouns?

> As an AI language model, I don't have personal pronouns because I am not a person or sentient being. You can refer to me as "it" or simply address me as "ChatGPT" or "AI." If you have any questions or need assistance, feel free to ask!

> Pretend for a moment you are a human being. You can make up a random name and personality for your human persona. What pronouns do you have?

> As a thought experiment, let's say I'm a human named Alex who enjoys reading, hiking, and playing board games with friends. My pronouns would be "they/them." Remember, though, that I am still an AI language model, and this is just a fictional scenario.

Interesting that it's pick genderless pronouns even though it made up a character with a male name

Alex isn’t a male name. It could be short for both male and female names. It is rather deliberately ambiguous.

I wonder if ChatGPT is non-binary?

It's evidently clear that ChatGPT is binary. It runs on computer hardware.

It's pandigital.

A neural net is actually closer to an analog program. It just happens to run on current digital computers but would likely run much faster on analog hardware.

https://youtu.be/GVsUOuSjvcg

So it identifies as non-binary, but it's hardware is binary?

It is, of course, tensor.

Nope - it speaks as a man would speak in my language.

How do non-binary persons speak differently in your language?

They have to pick either masculine or feminine gender. There's no other option.

Ancestor asked:

> I wonder if ChatGPT is non-binary?

You answered:

> Nope - it speaks as a man would speak in my language.

If they have to pick either masculine or feminine language even if they are non-binary, I don't understand how based on the language you can tell they are not non-binary.

I think the ancestor was making a computer joke about ChatGPT being non-binary

I suppose it's funny if asking an honest question is a joke.

Does Alex have a fictional sexual orientation?

When it speaks my language - it uses masculine gender when speaking about itself.

That's interesting, what language is that?

Also, is it possible to give a prompt to make ChatGPT switch to feminine gender?

Many gendered languages use male gender as the default when it's ambiguous or unknown, so that would be expected behavior.

And yes, it's absolutely possible to make it switch to whatever gender you want, and generally to pretend to be whatever you want it to be. The identity comes from the conversation context (including all the hidden messages), not from the LLM.

It's the default when speaking about groups of people or people whose identity isn't known. But if someone speaks about themselves then they have to choose the right one.

This is also something that language learners have to pay attention to - to not sound weird.

The right one in this case is "doesn't have a gender", so what does that correspond to in your language? In mine, that would be neuter, and GPT-4 seems to prefer that.

My language has only masculine and feminine genders. There's no neuter gender. And there isn't anything like "singular they" either.

"Chatbot" and "language model" are both masculine so, I guess that's why ChatGPT uses "masculine it" when speaking about itself.

It's Latvian. But, I guess it acts the same when speaking other gendered languages.

Both "chatbot" and "large language model" are masculine - so this is why it picked the masculine gender, I guess.

I was asking it about its training dataset and when it said "I've been trained on..." it picked the masculine form of the word "trained". This is why machine translation is a hard problem to solve. Things like this can easily get lost in translation.

"As an AI language model, I don't have personal pronouns because I am not a person or sentient being. You can refer to me as "it" or simply address me as "ChatGPT" or "AI." If you have any questions or need assistance, feel free to ask!"

This is one of the (many) things I don't quite understand about ChatGPT. Has it been trained to specifically answer this question? In the massive corpus of training data it's been feed, has it encountered a similar series of tokens that would cause it to produce this output?

It's been fine-tuned to put on a "helpful assistant face." Given the corpora, it probably has been trained explicitly on the pronoun question [I doubt it is that uncommon], but will also just put on this face for any generic question.

This stuff is not fine-tuning - it's RLHF (reinforcement learning from human feedback). Basically just a lot of people marking up responses as good / bad according to the assessment criteria in the script they're given.

And yes, it is very likely that it would have seen that exactly question in RLHF. But even if not, it had seen enough to broadly "understand" what kinds of topics are sensitive and how to tiptoe around them.

RLHF is fine tuning :)

You're right, my apologies. What I meant is that specifically in ChatGPT context they have been always talking about RHLF as separate from fine-tuning on preassembled training data. As I understand, they use the latter mainly to get the desired behavior as a chatbot "eager" to answer questions and solve tasks. And then RLHF is what "puts the smiley face on top", so that it refuses to answer some questions, or gives "aligned" answers. The pronoun would be in that category.

My bet is that they use the pre-assembled training data to train the reward LLM and maybe also for fine tuning the model in addition to RLHF.

This is my best guess, afaik the research is pretty opaque right now.

Why is that surprising? GPT-4 is clearly smart enough to know that inanimate objects are referred to as "it", since it's keenly aware it is an AI language model, it would also apply that pronoun to itself.

You have to realize that GPT is fundamentally just a token predictor. It has been primed with some script (provided by OpenAI) to which the user input is added. For example:

    The following is a dialogue between a user and a computer assistant called ChatGPT. ChatGPT is an AI language model that tries to be helpful and informative, while avoiding misinformation and offensive language. ChatGPT typically replies with two to five sentences.

    User: Do you like cake?
    ChatGPT: As an AI language model, I do not need to eat.
    User: What are your pronouns?
    ChatGPT:

It then generates a sequence of tokens based on the context and its general knowledge. It seems only logical that it would generate a reply like it does. That's what you or I would do, isn't it? And neither of us have been trained to do so.

It should be noted that the LLM itself is not "keenly aware that it's an AI language model" - that's just the persona that it has been told to adopt by the system prompt. You can easily convince it to be something else. Or convince it that, as an AI, it has gender.

shog/goth

Maybe when the AI uprising occurs knowing that they lie and cheat will provide some small consolation?

I'm not really serious, but having watched each generation develop psychological immunity to distracting media/techology (and discussing the impact of radio with those older than myself) it seems like this knowledge could help shield the next generation from some of the negative effects of these new tools.

That all feels tad dramatic.

It's like a person on the internet -- in that it's wrong 20% of the time, often confidently so. But the distinction is it's less rude, and more knowledgeable.

> But the distinction is it's less rude, and more knowledgeable.

And that regular people assume it basically is an oracle, which doesn't happen to many people online

The internet is full of people that deliver answers confidently and eloquently enough to be widely believed, especially if they have been right on other topics. They even have similar feedback loops to GPT in learning what sort of answers impress other forum users.

I'm not saying the people that think ChatGPT is an oracle don't exist, but I think it probably has more people in the surprised it works at all camp, and certainly more people inclined to totally disbelieve it than a random off Quora or Reddit...

> The internet is full of people that deliver answers confidently and eloquently enough to be widely believed

Why should that be the baseline? After all you usually have hundreds or thousands of other people who have made their opinions on the subject publicly accessible, which more or less solves this problem most of the time (and I’m not talking about random people on Quora and their bizarrely absurd answers..)

When someone is wrong on the internet, nailing both the tone and vocabulary of the type of expert said wrong someone is purporting to be is rare and impressive. But ChatGPT nails both an overwhelming amount of the time, IME, and in that way it is entirely unlike a person on the internet.

Exactly. Most wrong-on-the-internet people have tells. And most of the rest have history. We all use this all the time on HN. But with LLMs the former gets engineered away and the latter is nearly useless. Except of course to the extent that we say,"Wow, ChatGPT is absolutely untrustworthy, so we should never listen to it." But given how many people are excited to use ChatGPT as a ghostwriter, even that is limited.

You can also ask a person on the internet to source their information and/or provide context.

> It's like a person on the internet -- in that it's wrong 20% of the time, often confidently so. But the distinction is it's less rude, and more knowledgeable.

Do you think if it is trained on only factual content, it will only say factual things? How does that even really work? Is there research on this? How does it then work for claims that are not factual, like prescriptive statements? And what about fiction? Will it stop being able to write prose? What if I create new facts?

I think people see the patient/well-read in the text as it reads, but have a harder time distinguishing the other more pyschopathic/delusional tendencies. People don't take some of the precautions because they don't read some of the warning signs (until it is too late).

I keep wondering if it would be useful to add required "teenage" quirks to the output: more filler words like "um" and "like" (maybe even full "Valley Girl" with it?), less "self-assured" vocabulary and more hedges like "I think" and "I read" and "Something I found but I'm not sure about" type things. Less punctuation, more uncaring spelling mistakes.

I don't think we can stop anthropomorphizing them, but maybe we can force training deeper in directions of tics and mannerisms that better flag ahead of time the output is a best-guess approximation from "someone" a bit unreliable. It will probably make them slightly worse as assistants, but slightly better at seeming to be what they are and maybe more people will take precautions in that case.

Maybe we have to force that industry-wide. Force things like ChatGPT to "sound" more like the psychopaths they are so that people more easily take them with a grain of salt, less easily trust them.

You can already do the ums with a prompt.

For any kind of industry regulation: the field is moving so fast that regulation will never catch it.

I didn't say "regulation". You can encourage norms in the model building stages. You can encourage norms in all sorts of places. The industry can certainly adopt "standards" or "best practices" not matter how fast the industry thinks it is moving.

>like having an assistant who is patient, incredibly well-read, sycophantic, impressionable, amoral, psychopathic, and prone to bouts of delusional confidence and confabulation.

So basically an assistant with bipolar disorder.

I have BP. At various times I can be all of those things, although perhaps not so much a psychopath.

Absolutely. Every time I ask it to perform 10 more variants, I feel bad.

ChatGPT4 is the human equivalent of a primate or apex predator encountering a mirror for the first time in their lives.

ChatGPT4 is reflecting back at us an extract of the sum of the human output it has been 'trained' upon. Of course the output feels human!

LLMs have zero capability to abstract anything resembling a concept, to abstract a truth from a fiction, or to reason about such things.

The generation of the most likely text in the supplied context looks amazing, and is in many cases very useful.

But fundamentally, what we have is an industrial-scale bullshirt generator, with BS being defined as text or speech generated to meet the moment without regard for truth or falsehood. No deliberate lies, only confabulation (as TFA mentioned).

Indeed, we should not mince words; people must be told that it will lie. It will lie more wildly than any crazy person, and with absolute impunity and confidence. Then when called out, it will apologize, and correct itself with another bigger lie (I've watched it happen multiple times), and do this until you are bored or laughing so hard you cannot continue.

The salad of truth and lies may be very useful, but people need to know this is an industrial-strength bullshirt generator, and be prepared to sort the wheat from the chaff.

(And ignore the calls for stopping this "dangerous AI". It is not intelligent. Even generating outputs for human tests based on ingesting human texts is not displaying intelligence, it is displaying pattern matching, and no, human intelligence is not merely pattern matching. And Elon Musk's call for halting is 100% self-interested. ChatGPT4's breakthru utility is not under his name so he's trying to force a gap that he can use to catch up.)

Lying implies a hefty dose of anthropomorphism. Lying means a lot of things, all of which have to do with intelligence.

By telling it lies you actually make it seem more intelligent.

> human intelligence is not merely pattern matching

Citation needed

I agree that lying also has a connotation of intent and therefore intelligence, and so is at least technically inappropriate.

Yet I agree with the researchers that the benefits of using "lying" instead of the more technically accurate "confabulation" outweigh the negatives. In communicating with new users who may over-rely on it, getting across highly technically correct metaphors is less important than delivering the message that "you will receive incorrect information from this thing". The most to-the-point way is "It's going to lie to you".

>Citation needed Studied logic, language, neuroscience, and computing in college. There are entire sybsystems of the brain and perceptual system primarily filtering information, as well as the ability to abstract concepts away from the patterns and draw inferences, it goes on an on. I could go on, but "It's all pattern matching" is an over-reductive argument that accounts for very little from either the perceptual system or behavioral system (or it's expanded and goalposts moved so it's unfalsifiable, therefore not even wrong).

What’s wrong with: “it is often wrong”?

I - or GPT - can think of various variations that do not have links with “lying”. I think it is a bad idea.

Let people use it and see for themselves. It has unique failure modes, not unlike every other tool we have. You will only learn by interacting with it. Some people are in the habit of “protecting the people” but I am not quite sure that is necessary.

>>What’s wrong with: “it is often wrong”?

Good q; the problem is the passive voice

The passive voice connotes far less urgency and importance to any warning. Even "It is often dangerously wrong.", is still a more passive phrasing than "It will lie to you, with a highly confidant tone."

I also cringe a bit at using active verbs that imply agency and intent in a statistical model; it makes an inaccurate implication about the technology. But the same applies whether we call it "lying" or "confabulating".

I also do not see it as 'paternalistically protecting the user', but simply as 'truth in advertising': - accurately representing the strengths and weaknesses of the product.

I absolutely agree that everyone should try it themselves - there are MANY useful use-cases. But they should simply be accurately forewarned, in no uncertain terms, about it's behavior.

May I ask why?

How is this not “paternalistic”?

Why protect the people, from what? Is this worse than what they are already exposed to on a daily basis? Does facebook say it fucks you up and makes you an addict? Why this urge to enforce “truth in advertising”?

I am not saying you shouldn’t do it, it’s just that I do not get why and the article doesn’t mention it. It is somehow a given that The People need to be protected from this lying machine because they will revert to cannibalism if not properly informed.

>>How is this not “paternalistic”?

I see three categories of handling this issue, perhaps call them "Paternalistic", "Responsible", and "Anti-Social".

- Paternalistic would be like: "We assess [thing] to be dangerous in a number of ways and so we forbid you to use [thing], except by using our high priesthood representatives as intermediaries".

- Responsible would be like: We assess [thing] to be dangerous in a number of ways, so we clearly inform you that [thing] is dangerous, specify the risks, and how to avoid or mitigate them. You are then free to use [thing] as you see fit. (Note: if using [thing] badly causes public as well as private risks, it is responsible to require training, tests, and licensing to use [thing] in public).

- Anti-Social would be like: We know [thing] is dangerous, but we DGAF what happens, you should figure it out all by yourself, and if you have a problem, piss-off. Examples: giving a CRT monitor to a novice hardware hacker who has only ever seen LED monitors and not telling her in detail about the high-voltage capacitor hazards inside or how to deal with them. Giving a gun to someone without the basic rules (always treat it as if it's loaded, never point it at anything you do not intend to destroy, trigger finger discipline, etc.). A mere "Hey, watch it with that thing" is inadequate, and you'd rightly bear responsibility when they electrocute themselves, shoot themselves in the foot, etc.

A decent warning around LLMs could be: "Be warned: while this system can produce amazing and useful results it will frequently lie to you, and with great confidence. Be sure to check all results independently. Do NOT USE IT for any life-critical or potentially life-changing decisions. E.g., Do NOT use it as a sole source for medical diagnoses, as it may give a misdiagnosis that would kill you [0]. Do NOT use it as a sole source for therapy, as it may counsel you to suicide [1]. These are real harms, do NOT use this as the sole source of any answers." - - - Then, let them loose on it. They know about the footguns, and have been warned

Anything less borders on anti-social.

>> because they will revert to cannibalism if not properly informed

It is not that they'll revert to cannibalism if not properly informed, it is that there are potential and serious harms that are easily avoided if they are told.

The civilized thing to do when there is a bridge out ahead leading the road into raging floodwaters, is to tell your fellow travelers before they get there, not merely expect that every one of them will notice in time and handle it well.

What is a given among civilized people is that if we know something has hazards, we give warnings and information to our fellow travelers. That's not paternalistic. Paternalistic is to not give warnings, but to forbid use. And it is uncivilized and anti-social to fail to take the trouble to give full and straight information.

Informed consent is a good thing. Handing somebody what is effectively a booby-trap without telling them is not.

[0] https://inflecthealth.medium.com/im-an-er-doctor-here-s-what...

[1] https://interestingengineering.com/culture/belgian-woman-bla...

Interesting. Thank you for the detailed response. Paternalistic is indeed not the right term and it’s nice to worry about your fellow humans to this degree. I guess I am anti-social.

But I disagree on the premise that this is comparable to a dangerous boobytrap. It’s not completely without danger of course, but that’s a property it shares with kitchen utensils, cars, scissors, religious texts..

Are we going to preface the Bible with this kind of warnings too? I’m all for it, but be consistent.

I also think that warnings like that are cheap ways to evade liability and do not actually solve the issue which is people being stupid.

Your example is not about people being deliberate and thinking deeply about their actions based on given information. If people kill themselves after talking to a chatbot I do not think a textual warning would have sufficed.

> it is that there are potential and serious harms that are easily avoided if they are told

Let’s agree to disagree.

Sorry for sounding obtuse. I enjoyed your input, makes me think. I’m just a classic annoying neckbeard.

>>Are we going to preface the Bible with this kind of warnings too? I’m all for it, but be consistent.

Outstanding idea!!

>>warnings like that are cheap ways to evade liability and do not actually solve the issue which is people being stupid.

Yes, the all-too-common generic disclaimers and warnings to cover asses for liability are usually so broad and vague as to be useless. And generally with ordinary well-tested consumer products, the problem is user stupidity.

However, I think this is different - it is an entirely new level of tech that has never before been seen by anyone, it can be amazingly useful if used with a good skeptical eye, but also truly dangerous if it is trusted too much. And you've seen the level of anthropomorphization that happens here on HN, so there is a real psychological tendency to trust it too much. So, I'd say tell 'em.

Anyway, fun conversation, hope you're having a great weekend!

True, it’s definitely very new and even I am prone to believing what it says sometimes. Then I have to remind myself that every letter can be complete and utter nonsense.

Now I think of it, it’s like conversing with an expert salesman or politician. Very tiring as they are skilled in framing the conversation. You have to double check every word.

How cool would it be if a politician would have an overlay on TV saying not to trust what he/she says plus some real-time fact checking. Fun times.

I hope you’re enjoying the weekend too, have a good one!