News for hackers. Stuff that matters.
Re:Byte Latent Transformer: Patches Scale Better Than Tokens by psb217 on Saturday December 14, 2024 @02:26PM (#42418915)
One way of thinking about the "Approximate Monotonic Constraint" is that you're running a quick and dirty edge detector on the entropy. Ie, you're clipping based on the gradient of per-byte entropy wrt timestep compared to detecting an edge based on gradient of per-pixel intensity wrt pixel coordinates. It would be interesting to look at the raw sequences of per-byte entropies to see how strongly these sorts of "edges" correlate with human interpretable boundaries (words, prefixes, suffixes, etc).
Re:Byte Latent Transformer: Patches Scale Better Than Tokens by yorwba on Saturday December 14, 2024 @03:58PM (#42419390)
Figure 4 plots the entropy of each byte in "Daenerys Targeryen is in Game of Thrones, a fantasy epic by George R.R. Martin."