GigaToken: ~1000x faster Language model tokenization

Re:GigaToken: ~1000x faster Language model tokenization by rockwotj on Wednesday July 22, 2026 @09:04PM (#49015574)

but according to Little’s law, if you improve latency, you also improve throughput, right?

If I have the same number of CPU cores and they all can do their work in half the time they can double the number of requests now

[ Parent | on HN ]

Re:GigaToken: ~1000x faster Language model tokenization by scheme271 on Wednesday July 22, 2026 @09:20PM (#49015657)

I don't think that's accurate. If tokenization takes say 10ms and the rest of the inference steps take 50 ms then, improving tokenization will improve the time to first token but won't affect throughput much. After the first token, the inference steps effectively hide the tokenization time.

[ Parent | on HN ]

Re:GigaToken: ~1000x faster Language model tokenization by ludston on Thursday July 23, 2026 @04:18AM (#49018381)

It depends on whether or not latency is the constraint of throughput in your circumstances.

[ Parent | on HN ]