News for hackers. Stuff that matters.

A comment on GigaToken: ~1000x faster Language model tokenization. Parent comment

Re:GigaToken: ~1000x faster Language model tokenization by pipsterwo on Wednesday July 22, 2026 @03:03PM (#49011796)
1/1000 of inference compute is a non-trivial workload at scale. Gartner estimates ~$28B in inference spend for 2026 making this a $28 million dollar per year workload (edit: based on the assumption above)

Source: https://www.gartner.com/en/newsroom/press-releases/2026-07-2...

[ Parent | on HN ]
Re:GigaToken: ~1000x faster Language model tokenization by boroboro4 on Wednesday July 22, 2026 @03:30PM (#49012179)
The issue is it’s cpu compute which is underutilized in gpu clusters anyway, so practically it’s not really 1/1000.
[ Parent | on HN ]
Re:GigaToken: ~1000x faster Language model tokenization by pipsterwo on Wednesday July 22, 2026 @03:55PM (#49012487)
Totally, edited my comment to specify "based on the assumption above." The main takeaway I was going for was 0.1% is not a small number in this context
[ Parent | on HN ]