News for hackers. Stuff that matters.
Re:tiktoken: OpenAI’s Tokenizer by mark_l_watson on Friday December 16, 2022 @11:43AM (#34016692)
Any information on which human languages it works with? Some languages like Thai can be challenging so I am wondering how general this is.
Re:tiktoken: OpenAI’s Tokenizer by joelburget on Friday December 16, 2022 @12:19PM (#34017258)
It works on all human languages, just inefficiently. I ran it over a sample I found on wikipedia:

sample = "ฟองมันฟันหนู, ฟันหนูฟองมัน, ฝนทองฟองมัน" len(sample), len(enc.encode(sample))

This returns `39, 40` so it's just encoding one character at a time. It's probably like this for almost all non-English text.

Re:tiktoken: OpenAI’s Tokenizer by theragra on Friday December 16, 2022 @01:22PM (#34018127)
Yeah, at least it does it with Russian