You Should Probably Pay Attention to Tokenizers

Re:You Should Probably Pay Attention to Tokenizers by SEGyges on Wednesday October 23, 2024 @12:55PM (#41926909)

tokens are on average four characters and the number of residual streams (and therefore RAM) the LLM allocates to a given sequence is proportionate to the number of units of input. the flops is proportionate to their square in the attention calculation.

you can hypothetically try to ameliorate this by other means, but if you just naively drop from tokenization to character or byte level models this is what goes wrong

[ Parent | on HN ]

Re:You Should Probably Pay Attention to Tokenizers by p1esk on Thursday October 24, 2024 @01:31AM (#41932231)

4x seq length expansion doesn’t sound that bad.

[ Parent | on HN ]

Re:You Should Probably Pay Attention to Tokenizers by lechatonnoir on Thursday October 24, 2024 @05:03AM (#41933603)

I mean, it's not completely fatal, but it means an approximately 16x increase in runtime cost, if I'm not mistaken. That's probably not worth trying to solve letter counting in most applications.

[ Parent | on HN ]

Re:You Should Probably Pay Attention to Tokenizers by SEGyges on Friday October 25, 2024 @04:56PM (#41949679)

it is not necessarily 16x if you, e.g., decrease model width by a factor of 4 or so also, but yeah naively the RAM and FLOPs scale up by n^2

[ Parent | on HN ]