Re:Cerebras-GPT vs. LLaMA AI Model Performance Comparison
by spyder on Wednesday March 29, 2023 @05:30PM
(#35364002)
There was a paper where they have found that converting the numbers to scientific notation ( like 1.5e-7) has improved these transformer-based language models at math, if I remember correctly. (with a quick search I could not find the link to it now)
[ Parent
| on HN ]
Re:Cerebras-GPT vs. LLaMA AI Model Performance Comparison
by nl on Thursday March 30, 2023 @02:07AM
(#35368697)
Interesting.
It might be http://nlg.csie.ntu.edu.tw/~cjchen/papers/eacl2023.pdf
They show an increase from 65% to 70% on their "comparing numbers" benchmark.
Re:Cerebras-GPT vs. LLaMA AI Model Performance Comparison
by spyder on Thursday March 30, 2023 @05:24PM
(#35378992)
That's also an interesting one, but I think I found it, but it's a little different from what I remembered because it's not just the scientific notation that helps but more so the addition of positional tokens:
https://arxiv.org/abs/2102.13019
The essence of it from the abstract:
". In particular, the model fails to learn addition of five-digit numbers when using subwords (e.g., “32”), and it struggles to learn with character-level representations (e.g., “3 2”). By introducing position tokens (e.g., “3 10e1 2”), the model learns to accurately add and subtract numbers up to 60 digits. We conclude that modern pretrained language models can easily learn arithmetic from very few examples, as long as we use the proper surface representation."