Exactly. Currently LLMs use a fixed tokenizer which works best for English and (to a lesser degree) Western languages. The H-Net architecture doesn't use a fixed tokenizer, they use dynamicaly adaptive chunks of bytes, and (in a two-stage setup) chunks of chunks of bytes.

However, they (Cartesia) haven't yet published a technical report on their new version of H-Net, and it is unclear if they will.

It's not optimised deliberately for English, right? It's just that byte pair encoding naturally favours English and other Latin languages because they are dominant in the training data.

Even if you use bytes only, Unicode is biased towards English and other Latin languages, too.

They are typically optimized for English, for example, they often use rules for English contractions. See this thread: https://x.com/_albertgu/status/2113687798826111283

I don't know whether European languages work inherently better with tokenizers, but I don't see why they would.

logographic languages still win I guess, a 4-byte chinese character beats a n-character English word for that same concept.

but yeah, anything alphabetic yet not latin gets the worst tradeoff.

Most CJK characters are 3 bytes in UTF-8. But then most English words translate to more than one character, so 5 bytes might be more realistic (apparently Chinese averages 1.6-1.7 characters per word).

> It's just that byte pair encoding naturally favours English and other Latin languages because they are dominant in the training data.

Not really, because I don't think anyone use the whole training dataset to train the tokenizer.

From my brief (disclosure: claude-assisted [0]) internet search I learned that typically they use from tens to hundreds GB of data to train the tokenizer. Sometimes it is hand-curated, sometimes it is randomly sampled.

Of course, you can deliberately choose how much text in each language do you include in the training set for the BPE and deliberately optimize for (or against) a specific language.

[0] https://claude.ai/share/da277189-f23a-4071-bd9f-2d75fe6aa779

Ive always thought that Serbian is the worst off out of all European languages. Both cyrillic and latin script is widely used. Latin also has a naked variant, where all diacritics are left out (which is especially prevalent online). Also, it is non-EU, so initiatives like [0] will only indirectly help with representation (croatian being the same language, but it does not use cyrillic script)

[0] https://digital-strategy.ec.europa.eu/en/news/commission-wel...

[delayed]

It is desirable for all alphabets/abjads/abugidas/syllabaries, not just Latin. Chinese is rather the special case of a human language with very high character inventory and hence low character/word ratio.

Non-latin languages also take more bytes due to utf-8 though

Are bytes relevant? These things are billed by the megatoken, not the megabyte.

If tokens use UTF-8 bytes then it could be a factor.

A token is an index into a dictionary. What the dictionary uses isn't seen, this is why they couldn't count the 'r's in 'strawberry'.

I was thinking both of input and latent data. Hadn't thought of tokens that way. Thanks for the heads up.

H net directly operates on bytes. This is what the comment was about and what I was referring to.

utf8 is just one of the encodings that exist, and is kind of irrelevant at the point it contacts the llm

No, the model is trained on utf-8, so switching encodings entails a full retrain

[delayed]

I've done my own research on that as a part of my Aqueous [1] archiver, and the conclusion is that the inherent information entropy is about as same (CJK scripts, for example, compresses much better when they are turned into their native character encodings). So a script-aware tokenization should make that irrelevant.

[1] https://lifthrasiir.github.io/aqueous/ (specifically wrt-recons-hangul codec)

The models the top post refers to are not script aware. This is my point

Non-latin characters use more bytes due to utf-8. But texts don't necessarily use more bytes because of that. I'd be surprised if, for example mandarin, uses significantly more bytes on average.

This is incidental and a cherry pick. Cyrillic and greek are regular alphabets, and cost two bytes per character instead of one.