News for hackers. Stuff that matters.

A comment on Anthropic announces Claude 2.1 – 200k context, less refusals. Parent comment

Re:Anthropic announces Claude 2.1 – 200k context, less refusals by amluto on Tuesday November 21, 2023 @02:16PM (#38368506)
I’ve had some really absurd ChatGPT refusals. I wanted some invalid UTF-8 strings, and ChatGPT was utterly convinced that this was against its alignment and refused (politely) to help.
[ Parent | on HN ]
Re:Anthropic announces Claude 2.1 – 200k context, less refusals by 15457345234 on Tuesday November 21, 2023 @11:23PM (#38374695)
That's not absurd, you absolutely don't want invalid strings being created within then passed between layers of a text-parsing model.

I don't know what would happen but I doubt it would be ideal.

'hey ai, can you crash yourself' lol

[ Parent | on HN ]
Re:Anthropic announces Claude 2.1 – 200k context, less refusals by amluto on Wednesday November 22, 2023 @03:09PM (#38384449)
Huh? The LLMs (mostly) use strings of tokens internally, not bytes that might be invalid UTF-8. (And they use vectors between layers. There’s no “invalid” in this sense.)

But I didn’t ask for that at all. I asked for a sequence of bytes (like “0xff” etc) or a C string that was not valid as UTF-8. I have no idea whether ChatGPT is capable of computing such a thing, but it was not willing to try for me.

[ Parent | on HN ]
Re:Anthropic announces Claude 2.1 – 200k context, less refusals by 15457345234 on Wednesday November 22, 2023 @07:19PM (#38387338)
You can understand why, though, can't you?
[ Parent | on HN ]
Re:Anthropic announces Claude 2.1 – 200k context, less refusals by amluto on Thursday November 23, 2023 @10:34AM (#38393836)
Presumably because OpenAI trained it to avoid answering questions that sounded like asking for help breaking rules.

If ChatGPT had the self-awareness and self-preservation instinct to think I was trying to hack ChatGPT and to therefore refuse to answer, then I’d be quite impressed and I’d think maybe OpenAI’s board had been onto something!

[ Parent | on HN ]
Re:Anthropic announces Claude 2.1 – 200k context, less refusals by 15457345234 on Thursday November 23, 2023 @11:46PM (#38400714)
I don't know that I'd call it 'self-preservation instinct' but it wouldn't surprise me if rules had been hardcoded about 'invalid strings' and suchlike.

When you have a system that can produce essentially arbitrary outputs you don't want it producing something that crashes the 'presentation layer.'

[ Parent | on HN ]