Re:Byte Latent Transformer: Patches Scale Better Than Tokens
by PaulHoule on Saturday December 14, 2024 @09:25AM
(#42417331)
I've often wanted to talk with an LLM about its tokenization (e.g. how many tokens are there in "the simplest of phrases") I wonder if you fed it information about its tokenization (text like "rabbit is spelled r, a, b, b, i, t") if it could talk about it.
[ Parent
| on HN ]
There are no comments on this comment yet.