Re:Cerebras-GPT vs. LLaMA AI Model Performance Comparison
by PeterisP on Thursday March 30, 2023 @04:41AM
(#35369761)
The general tokenization screws up performance on some very specific niche tasks, which are not used in training or evaluation, but are used a 'toy examples' for people to test the system, resulting in stupid outcomes that are treated as informative of the model's general performance.
I.e. "how many letters are in a word" or "which is the fifth letter of this word" type tasks is screwed up because the model has no information about the length of each token other than what it can infer from seeing some words in similar challenges in children learning materials.
"does 123912879212392 contain the digit 7" can fail because the model has limited information that the token "79" in the middle is somehow related to the token "7".
"is 50100 greater than 50200" becomes a tricky challenge because due to its tokenization it tries to compare "501"/"00" with "50"/"200" while knowing that "501" is greater than "50".
I mean, all of it can be learned if given sufficient data, however, we shouldn't be surprised if some tasks seem more difficult than they should be if we have explicitly crippled the representation we use to communicate these tasks to the system.
I.e. "how many letters are in a word" or "which is the fifth letter of this word" type tasks is screwed up because the model has no information about the length of each token other than what it can infer from seeing some words in similar challenges in children learning materials.
"does 123912879212392 contain the digit 7" can fail because the model has limited information that the token "79" in the middle is somehow related to the token "7".
"is 50100 greater than 50200" becomes a tricky challenge because due to its tokenization it tries to compare "501"/"00" with "50"/"200" while knowing that "501" is greater than "50".
I mean, all of it can be learned if given sufficient data, however, we shouldn't be surprised if some tasks seem more difficult than they should be if we have explicitly crippled the representation we use to communicate these tasks to the system.