News for hackers. Stuff that matters.
Re:Transformers Can Do Arithmetic with the Right Embeddings by psychoslave on Tuesday May 28, 2024 @06:21AM (#40499201)
I didn’t test with all LLM out there, but all of thus I tested failed with something as basic as "What is the number of words in the sentence coming before the next one? Please answer."
Re:Transformers Can Do Arithmetic with the Right Embeddings by asgeir on Tuesday May 28, 2024 @07:08AM (#40499502)
In my experience, LLMs tend to perform better if you give them instructions before the data to be operated on. At least for the ~13b size models.

So,something like: Please count the number of words in the following sentence. "What is the number of words in the sentence coming before the next one?"

edit: Which might be an artifact of the training data always being in that kind of format.

Re:Transformers Can Do Arithmetic with the Right Embeddings by olalonde on Tuesday May 28, 2024 @07:28AM (#40499638)
GPT-4 (OpenAI):

The sentence you're referring to is "What is the number of words in the sentence coming before the next one? Please answer." It contains 14 words.

Re:Transformers Can Do Arithmetic with the Right Embeddings by psychoslave on Tuesday May 28, 2024 @08:54AM (#40500268)
Thanks. I don’t have access to this engine which for some reason is kept in a closed garden for richer people. ¯\_(ツ)_/¯
Re:Transformers Can Do Arithmetic with the Right Embeddings by wrsh07 on Tuesday May 28, 2024 @10:23AM (#40501019)
You can always use the API which is dirt cheap? Just put $5 on and access via the playground

They have better data policies and your $5 will go way farther than a 1 month subscription

Re:Transformers Can Do Arithmetic with the Right Embeddings by mikeocool on Tuesday May 28, 2024 @11:34AM (#40501752)
Interestingly, chat gpt 4o gave me the answer 15.
Re:Transformers Can Do Arithmetic with the Right Embeddings by itchyjunk on Tuesday May 28, 2024 @06:41AM (#40499309)
How many humans have you tested this with?
Re:Transformers Can Do Arithmetic with the Right Embeddings by psychoslave on Tuesday May 28, 2024 @06:48AM (#40499350)
Interesting point. Would you please answer the question I was mentioning? :)
Re:Transformers Can Do Arithmetic with the Right Embeddings by Thorham on Tuesday May 28, 2024 @07:09AM (#40499508)
14