News for hackers. Stuff that matters.
Re:Byte Latent Transformer: Patches Scale Better Than Tokens by PaulHoule on Saturday December 14, 2024 @09:25AM (#42417331)
I've often wanted to talk with an LLM about its tokenization (e.g. how many tokens are there in "the simplest of phrases") I wonder if you fed it information about its tokenization (text like "rabbit is spelled r, a, b, b, i, t") if it could talk about it.

There are no comments on this comment yet.