News for hackers. Stuff that matters.

A comment on You Should Probably Pay Attention to Tokenizers. Parent comment

Re:You Should Probably Pay Attention to Tokenizers by cma on Wednesday October 23, 2024 @03:20PM (#41928442)
Context length performance and memory scales N^2. Smaller tokens mean worse scaling, up to a point.
[ Parent | on HN ]

There are no comments on this comment yet.