News for hackers. Stuff that matters.

A comment on GigaToken: ~1000x faster Language model tokenization. Parent comment

Re:GigaToken: ~1000x faster Language model tokenization by antonvs on Thursday July 23, 2026 @01:57AM (#49017436)
> … only for the run to crash over some minor bug after that

Ah, Python.

[ Parent | on HN ]
Re:GigaToken: ~1000x faster Language model tokenization by janalsncm on Thursday July 23, 2026 @04:47AM (#49018612)
An example would be setting batch size too high causing OOM which isn’t really a python problem.
[ Parent | on HN ]