News for hackers. Stuff that matters.

A comment on GigaToken: ~1000x faster Language model tokenization. Parent comment

Re:GigaToken: ~1000x faster Language model tokenization by fastball on Wednesday July 22, 2026 @04:18PM (#49012818)
Tokenization is <0.1% of the inference time for the first token in the same way it is <0.1% for the last.
[ Parent | on HN ]
Re:GigaToken: ~1000x faster Language model tokenization by marcelroed on Wednesday July 22, 2026 @04:57PM (#49013327)
Time to first token refers to the time until the model outputs one token, which includes the time to process the entire prompt (doing prefill). The GPU time per token is much lower when doing prefill, so the significance of tokenization is higher.
[ Parent | on HN ]
Re:GigaToken: ~1000x faster Language model tokenization by dingdingdang on Wednesday July 22, 2026 @06:13PM (#49014179)
Have you done preliminary numbers on replacing tokenizer on, say, llama-server?
[ Parent | on HN ]
Re:GigaToken: ~1000x faster Language model tokenization by marcelroed on Wednesday July 22, 2026 @07:44PM (#49015028)
Added numbers here: https://news.ycombinator.com/item?id=49015014
[ Parent | on HN ]
Re:GigaToken: ~1000x faster Language model tokenization by marcelroed on Wednesday July 22, 2026 @07:01PM (#49014640)
Running the numbers now
[ Parent | on HN ]