Every story and comment from the Hacker News API, in a layout borrowed from 2001.
A comment on tiktoken: OpenAI’s Tokenizer. Parent comment
sample = "ฟองมันฟันหนู, ฟันหนูฟองมัน, ฝนทองฟองมัน" len(sample), len(enc.encode(sample))
This returns `39, 40` so it's just encoding one character at a time. It's probably like this for almost all non-English text.
sample = "ฟองมันฟันหนู, ฟันหนูฟองมัน, ฝนทองฟองมัน" len(sample), len(enc.encode(sample))
This returns `39, 40` so it's just encoding one character at a time. It's probably like this for almost all non-English text.