Re:Byte Latent Transformer: Patches Scale Better Than Tokens
by ted_dunning on Saturday December 14, 2024 @04:50PM
(#42419701)
It is also quite similar to Carl de Marcken's work for segmenting text and speech. He phrased everything in terms of minimum description length (MDL), but that is trivially the same thing as local entropy.
https://dspace.mit.edu/handle/1721.1/7191?show=full