News for hackers. Stuff that matters.
Re:Gemini Ultra Released by planb on Thursday February 08, 2024 @09:51AM (#39302560)
Please anyone correct me if I'm wrong: LLMs cannot solve this kind of riddle. This has nothing to do with their capabilities for logical reasoning, but with the way words are represented as tokens. While they might know that "apples" has two syllables because that is mentioned somewhere in their training data, if you make up a fruit "bratush" a human will see that as two syllables, but this might be 1 to 7 tokens to a LLM without any information about the word itself.
Re:Gemini Ultra Released by andrewla on Thursday February 08, 2024 @12:27PM (#39304786)
The amazing thing about emergent behavior in LLMs is that they are able to answer questions like these. I don't think it is completely understood how exactly they do this, but there's little doubt that they do.
Re:Gemini Ultra Released by boppo1 on Thursday February 08, 2024 @01:01PM (#39305250)
Do you have any sources that prove this is true?
Re:Gemini Ultra Released by jerbear4328 on Thursday February 08, 2024 @01:22PM (#39305522)
Re:Gemini Ultra Released by bitmovements on Friday February 09, 2024 @10:47AM (#39315982)
Your word is not made up, nor are some of the others in the sample like”pez.” I don’t think this test proves what you think it does. Bratush come up quite a bit in the internet: https://www.thefreedictionary.com/words-that-start-with-brat...
Re:Gemini Ultra Released by ComputerGuru on Thursday February 08, 2024 @10:27AM (#39303010)
GPT-4 has a really good tokenizer that is able to retain and use more information about input tokens than one might naively think.
Re:Gemini Ultra Released by planb on Thursday February 08, 2024 @11:04AM (#39303550)
Interesting. Given the example with made up words someone posted here it really looks like I was wrong.
Re:Gemini Ultra Released by huytersd on Thursday February 08, 2024 @10:12AM (#39302818)
Well I tried it out in GPT4 with made up words-

Tabitha likes bratush but not zot. She likes protel but not kig, and she likes motsic but not pez. Following the same rule, will she like tridos or kip

Given the examples, one speculative pattern could be that Tabitha likes words with at least two syllables or a certain complexity in structure. Therefore, following this speculative rule, Tabitha might like “tridos” more than “kip.”

Re:Gemini Ultra Released by bitmovements on Friday February 09, 2024 @10:56AM (#39316100)
Zot is a word already being used in the world: verb. (slang) To zap, kill, or destroy.

So is protel: https://en.m.wikipedia.org/wiki/Protel

So is kig: https://en.m.wiktionary.org/wiki/kig

Pez is a well known brand name in America.

Kip is commonly used name in America.

Motsic is a fairly common last name from searching.

Tridos is used all over the internet as a brand name so this all seems probable to be in the training data.

These words are not new nor are they made up.

Re:Gemini Ultra Released by mborch on Thursday February 08, 2024 @10:10AM (#39302805)
ChatGPT knows how to answer this question, but how I don't know. Perhaps it's programmed to answer that question?
Re:Gemini Ultra Released by viraptor on Thursday February 08, 2024 @02:18PM (#39306259)
LLM can solve this for all tokens where it got to learn how many syllables are in that token or a combination. If you trained it to work on single letters only it would do better at that task than word chunks (same for math and single digits). It will generalise to new words if the token level knowledge is there.

Whether this means it can or cannot solve that kind of riddle is up for your interpretation. I understand square root and can calculate square root of 16, but not of 738284.7280594873. (in a reasonable, bounded time) Can I solve square roots?