News for hackers. Stuff that matters.

A comment on Byte Latent Transformer: Patches Scale Better Than Tokens. Parent comment

Re:Byte Latent Transformer: Patches Scale Better Than Tokens by cschmidt on Saturday December 14, 2024 @02:46PM (#42418998)
I'm curious if you're aware of some papers from around 2005 on using contextual entropy to do unsupervised word segmentation on Chinese, and other languages that don't use spaces for word boundaries.

https://aclanthology.org/Y03-1017/ https://aclanthology.org/I05-1009/ https://aclanthology.org/P06-2056/

Exactly the same approach of segmenting a word when the entropy goes up compared to the previous byte.

[ Parent | on HN ]
Re:Byte Latent Transformer: Patches Scale Better Than Tokens by entilzha on Saturday December 14, 2024 @03:05PM (#42419097)
At least I wasn't aware of this work, but thanks for the refs! I'm always curious to read papers from 10-20+ years ago that have similarly inspired ideas. If it makes sense, we'll mention those in the next related work update.
[ Parent | on HN ]
Re:Byte Latent Transformer: Patches Scale Better Than Tokens by ted_dunning on Saturday December 14, 2024 @04:50PM (#42419701)
It is also quite similar to Carl de Marcken's work for segmenting text and speech. He phrased everything in terms of minimum description length (MDL), but that is trivially the same thing as local entropy.

https://dspace.mit.edu/handle/1721.1/7191?show=full

[ Parent | on HN ]