News for hackers. Stuff that matters.

A comment on Byte Latent Transformer: Patches Scale Better Than Tokens.

Re:Byte Latent Transformer: Patches Scale Better Than Tokens by amelius on Saturday December 14, 2024 @09:03AM (#42417206)
Why can't the tokenization be implicit, so we only feed bytes (or characters) to the model?
[ Parent | on HN ]
Re:Byte Latent Transformer: Patches Scale Better Than Tokens by entilzha on Saturday December 14, 2024 @09:43AM (#42417428)
(Author Here)

Not sure what you mean by implicit? If you mean just treat bytes as tokens, one issue you run into is your sequence lengths get quite long, so compared to a regular token LLM, you can’t pack as many bytes in a batch, which means you’re pretty FLOP inefficient so scale worse. You could make the model smaller to compensate, but then the model isn’t as good.

[ Parent | on HN ]
Re:Byte Latent Transformer: Patches Scale Better Than Tokens by killerstorm on Saturday December 14, 2024 @09:42AM (#42417423)
It can work, but you have more tokens / weaker performance.

People tested it and it was worse.

[ Parent | on HN ]