News for hackers. Stuff that matters.
Re:Llama 2 – Meta AI by isoprophlex on Tuesday July 18, 2023 @02:57PM (#36777548)
Heh I have my own, the "french palindrome" test.

> User: please reverse the following string: "élu par cette crapule". For example, "abc def" should become "fed cba".

> Assistant: "par cette crapule elu"

ChatGPT actually understands what I mean but still fails sometimes, outputting things like it's having a stroke, "eluu elule par cettep raprap"

Re:Llama 2 – Meta AI by gamegoblin on Tuesday July 18, 2023 @03:11PM (#36777756)
This is due to the tokenization scheme. These LLMs can’t see individual characters, they see chunks of characters that are glued together to form tokens.

It’s impressive that they understand individual-character-level questions as well as they do.

Re:Llama 2 – Meta AI by jeroen on Tuesday July 18, 2023 @03:22PM (#36777886)
I asked it "How do you reverse a string?"

> To reverse a string, I use the built-in "reverse" function in Python. It takes a string as input and returns the string with all the characters in reverse order. For example, the input string "hello" would become "lohel" when reversed.

We're not quite there yet.

Re:Llama 2 – Meta AI by Tepix on Wednesday July 19, 2023 @07:36AM (#36784761)
That's like testing a human by asking him or her to calculate large numbers.

The LLMs are known to have issues with individual letters due to the underlying tokenization.