Re:Transformers Can Do Arithmetic with the Right Embeddings
by psychoslave on Tuesday May 28, 2024 @06:21AM
(#40499201)
I didn’t test with all LLM out there, but all of thus I tested failed with something as basic as "What is the number of words in the sentence coming before the next one? Please answer."
[ Parent
| on HN ]
Re:Transformers Can Do Arithmetic with the Right Embeddings
by asgeir on Tuesday May 28, 2024 @07:08AM
(#40499502)
In my experience, LLMs tend to perform better if you give them instructions before the data to be operated on. At least for the ~13b size models.
So,something like: Please count the number of words in the following sentence. "What is the number of words in the sentence coming before the next one?"
edit: Which might be an artifact of the training data always being in that kind of format.
Re:Transformers Can Do Arithmetic with the Right Embeddings
by olalonde on Tuesday May 28, 2024 @07:28AM
(#40499638)
GPT-4 (OpenAI):
The sentence you're referring to is "What is the number of words in the sentence coming before the next one? Please answer." It contains 14 words.
Re:Transformers Can Do Arithmetic with the Right Embeddings
by psychoslave on Tuesday May 28, 2024 @08:54AM
(#40500268)
Thanks. I don’t have access to this engine which for some reason is kept in a closed garden for richer people. ¯\_(ツ)_/¯
Re:Transformers Can Do Arithmetic with the Right Embeddings
by wrsh07 on Tuesday May 28, 2024 @10:23AM
(#40501019)
You can always use the API which is dirt cheap? Just put $5 on and access via the playground
They have better data policies and your $5 will go way farther than a 1 month subscription
Re:Transformers Can Do Arithmetic with the Right Embeddings
by mikeocool on Tuesday May 28, 2024 @11:34AM
(#40501752)
Interestingly, chat gpt 4o gave me the answer 15.
Re:Transformers Can Do Arithmetic with the Right Embeddings
by itchyjunk on Tuesday May 28, 2024 @06:41AM
(#40499309)
How many humans have you tested this with?
Re:Transformers Can Do Arithmetic with the Right Embeddings
by psychoslave on Tuesday May 28, 2024 @06:48AM
(#40499350)
Interesting point. Would you please answer the question I was mentioning? :)