Re:GigaToken: ~1000x faster Language model tokenization
by marcelroed on Wednesday July 22, 2026 @04:57PM
(#49013327)
Time to first token refers to the time until the model outputs one token, which includes the time to process the entire prompt (doing prefill). The GPU time per token is much lower when doing prefill, so the significance of tokenization is higher.
[ Parent
| on HN ]
Re:GigaToken: ~1000x faster Language model tokenization
by dingdingdang on Wednesday July 22, 2026 @06:13PM
(#49014179)
Have you done preliminary numbers on replacing tokenizer on, say, llama-server?
Re:GigaToken: ~1000x faster Language model tokenization
by marcelroed on Wednesday July 22, 2026 @07:44PM
(#49015028)
Added numbers here: https://news.ycombinator.com/item?id=49015014
Re:GigaToken: ~1000x faster Language model tokenization
by marcelroed on Wednesday July 22, 2026 @07:01PM
(#49014640)
Running the numbers now