News for hackers. Stuff that matters.
Re:Tokenisation Is NP-Complete by Vecr on Monday December 23, 2024 @02:48AM (#42492632)
It's not optimal, but if you want to play it safe you can use 1 byte = 1 token, plus a few control tokens like stop, user turn, model turn, etc.

There are no comments on this comment yet.