News for hackers. Stuff that matters.
Re:Byte Latent Transformer: Patches Scale Better Than Tokens by entilzha on Saturday December 14, 2024 @09:49AM (#42417463)
(Author Here)

In editing we couldn’t find a good place for this so cut it in the current version, but at one point had discussed a parallel with information density of speech as described by one paper. Essentially the paper found that in languages that were less information dense per syllable, speakers spoke faster to achieve similar information density as languages with higher density per syllable. You could see patching by entropy paralleling this if you consider that low entropy bytes in terms of Shannon entropy are less information dense.

There are no comments on this comment yet.