Are bytes relevant? These things are billed by the megatoken, not the megabyte.

If tokens use UTF-8 bytes then it could be a factor.

A token is an index into a dictionary. What the dictionary uses isn't seen, this is why they couldn't count the 'r's in 'strawberry'.

I was thinking both of input and latent data. Hadn't thought of tokens that way. Thanks for the heads up.

H net directly operates on bytes. This is what the comment was about and what I was referring to.

utf8 is just one of the encodings that exist, and is kind of irrelevant at the point it contacts the llm

No, the model is trained on utf-8, so switching encodings entails a full retrain

[delayed]

I've done my own research on that as a part of my Aqueous [1] archiver, and the conclusion is that the inherent information entropy is about as same (CJK scripts, for example, compresses much better when they are turned into their native character encodings). So a script-aware tokenization should make that irrelevant.

[1] https://lifthrasiir.github.io/aqueous/ (specifically wrt-recons-hangul codec)

The models the top post refers to are not script aware. This is my point

Non-latin characters use more bytes due to utf-8. But texts don't necessarily use more bytes because of that. I'd be surprised if, for example mandarin, uses significantly more bytes on average.

This is incidental and a cherry pick. Cyrillic and greek are regular alphabets, and cost two bytes per character instead of one.