Re:Gemma: New Open Models
by espadrine on Wednesday February 21, 2024 @12:45PM
(#39456972)
It mostly means that there are tokens dedicated to rarer sequences of characters, even in foreign languages (note that Gemma is not intended to be good multilingually): “説明書” (instruction manual) has its own token, and so does “Nixon”, “آباد” (a city suffix, I believe), and the HTML sequence "\"><!--".
[ Parent
| on HN ]
Re:Gemma: New Open Models
by lalaithion on Wednesday February 21, 2024 @02:27PM
(#39458319)
I understand the theory, I was looking for an example of the same text tokenized with the two different vocabularies.
Do you have an example text in mind?
You can use this playground to test it out: https://huggingface.co/spaces/Xenova/the-tokenizer-playgroun...
Re:Gemma: New Open Models
by moffkalast on Thursday February 22, 2024 @04:11AM
(#39464851)
Interesting, it's actually worse than GPT-4s 100k tokenizer by quite a bit despite being over twice the size and only marginally better than LLama's 30k. At least for some random articles in English latin script that I tried anyway, but Llama and Gemma are English-only models so no point in testing anything else.
Doesn't seem like a well made tokenizer at first glance or it's heavily biased towards languages the model can't even generate coherently, lol. If they really wanted it to be SOTA at something they could've at least made it the first open source truly multilingual model, but that's apparently more effort than the lame skin colour oriented virtue signalling Google wants to do.