News for hackers. Stuff that matters.

A comment on Cerebras-GPT vs. LLaMA AI Model Performance Comparison. Parent comment

Re:Cerebras-GPT vs. LLaMA AI Model Performance Comparison by gerad on Wednesday March 29, 2023 @07:44PM (#35365596)
I think what OP means is that the tokenization is throwing away encoded information.

That is 11 should be two tokens representing 1 and 10. That way the relationship can be more easily learned.

Otherwise it needs to memorize the relationship between every pair of numbers. Which is exponentially more difficult.

[ Parent | on HN ]
Re:Cerebras-GPT vs. LLaMA AI Model Performance Comparison by nl on Wednesday March 29, 2023 @07:59PM (#35365758)
Yes this is a fair point, and if that is what they meant I could agree it is worth trying. As pointed out down-thread the Llama tokenizer is supposed to spilt numbers into their digits.
[ Parent | on HN ]