Re:Ts_zip: Text Compression Using Large Language Models
by londons_explore on Wednesday August 16, 2023 @06:02PM
(#37154480)
Even if you trained an LLM with a simplified tokenizer that simply had 256 tokens for each of 256 possible ascii characters, you would see the same result.
LLMs are sort of unable to do this because they use a fixed tokenizer instead of raw bytes. That means they won't output binary garbage even early on + saves a lot of memory, but it may hurt learning things like capitalization, rhyming, etc we think are obvious.