Re:Byte Latent Transformer: Patches Scale Better Than Tokens
by psb217 on Saturday December 14, 2024 @02:26PM
(#42418915)
One way of thinking about the "Approximate Monotonic Constraint" is that you're running a quick and dirty edge detector on the entropy. Ie, you're clipping based on the gradient of per-byte entropy wrt timestep compared to detecting an edge based on gradient of per-pixel intensity wrt pixel coordinates. It would be interesting to look at the raw sequences of per-byte entropies to see how strongly these sorts of "edges" correlate with human interpretable boundaries (words, prefixes, suffixes, etc).
[ Parent
| on HN ]