Re:Byte Latent Transformer: Patches Scale Better Than Tokens
by flimflamm on Saturday December 14, 2024 @06:18AM
(#42416323)
"That's not how it's described" - Thanks for the correction!
Re:Byte Latent Transformer: Patches Scale Better Than Tokens
by entilzha on Saturday December 14, 2024 @10:12AM
(#42417582)
(Author Here)
Good description! Maybe what parent got mixed up on is an alternate way to view this is trying to chunk bytes to have roughly similar information. EG we initially tried a bunch of patching schemes, EG, keep a running total of entropy until the total exceeds a threshold, but ended up finding simple things worked better.
I’ll see if we can add more information about the small CNN in a next update to arXiv paper.
Re:Byte Latent Transformer: Patches Scale Better Than Tokens
by psb217 on Saturday December 14, 2024 @02:26PM
(#42418915)
One way of thinking about the "Approximate Monotonic Constraint" is that you're running a quick and dirty edge detector on the entropy. Ie, you're clipping based on the gradient of per-byte entropy wrt timestep compared to detecting an edge based on gradient of per-pixel intensity wrt pixel coordinates. It would be interesting to look at the raw sequences of per-byte entropies to see how strongly these sorts of "edges" correlate with human interpretable boundaries (words, prefixes, suffixes, etc).
Re:Byte Latent Transformer: Patches Scale Better Than Tokens
by yorwba on Saturday December 14, 2024 @03:58PM
(#42419390)
Figure 4 plots the entropy of each byte in "Daenerys Targeryen is in Game of Thrones, a fantasy epic
by George R.R. Martin."
Re:Byte Latent Transformer: Patches Scale Better Than Tokens
by cschmidt on Saturday December 14, 2024 @02:46PM
(#42418998)
I'm curious if you're aware of some papers from around 2005 on using contextual entropy to do unsupervised word segmentation on Chinese, and other languages that don't use spaces for word boundaries.
https://aclanthology.org/Y03-1017/ https://aclanthology.org/I05-1009/ https://aclanthology.org/P06-2056/
Exactly the same approach of segmenting a word when the entropy goes up compared to the previous byte.
Re:Byte Latent Transformer: Patches Scale Better Than Tokens
by entilzha on Saturday December 14, 2024 @03:05PM
(#42419097)
At least I wasn't aware of this work, but thanks for the refs! I'm always curious to read papers from 10-20+ years ago that have similarly inspired ideas. If it makes sense, we'll mention those in the next related work update.
Re:Byte Latent Transformer: Patches Scale Better Than Tokens
by ted_dunning on Saturday December 14, 2024 @04:50PM
(#42419701)
It is also quite similar to Carl de Marcken's work for segmenting text and speech. He phrased everything in terms of minimum description length (MDL), but that is trivially the same thing as local entropy.
That's not how it's described in Section 2.3 of the paper. They only use the entropy of the next byte and whether it exceeds a threshold (Global Constraint) or is larger than the preceding byte's entropy by another threshold (Approx. Monotonic Constraint).
That does mean that long repetitive sequences can result in pathologically long patches, as demonstrated in Appendix E.
But what I'm really curious about is the "small CNN byte-level model with 2-byte context" in Figure 3 (f), because it's never mentioned in any other part of the paper.