Re:The Limits of GPT-3 and Similar Large Language Models
by stared on Saturday January 07, 2023 @03:51PM
(#34292640)
Tokenization for models like GPT or BERT can be seen as compression. That is, frequent words are separate tokens. Frequent sequences are separate tokens. On the other hand, if a sequence is very uncommon, then it will contain many tokens.
Sure, you encode bit-by-bit. But it is a fixed-length code, which is even worse than character-by-character.
Maybe you only get worse training and inference time. But I wouldn't be surprised if the encoding also serves as a Bayesian prior, and with a different encoding, you get worse results (for given data).
Sure, you encode bit-by-bit. But it is a fixed-length code, which is even worse than character-by-character.
Maybe you only get worse training and inference time. But I wouldn't be surprised if the encoding also serves as a Bayesian prior, and with a different encoding, you get worse results (for given data).