Every story and comment from the Hacker News API, in a layout borrowed from 2001.
A comment on Why is GPT-3 15.77x more expensive for certain languages?.
Author compares different encoders: for Facebook's NLLB and GPT2. Where did title came from?
Another point is that OpenAI changed encoders for chat models. Link: https://github.com/openai/openai-cookbook/blob/main/examples...
Now English is less optimized for tokens usage and other languages are much more balanced. E.g. Ukrainian takes only twice as much tokens, before it had 6 times more tokens
There are no comments on this comment yet.
Author compares different encoders: for Facebook's NLLB and GPT2. Where did title came from?
Another point is that OpenAI changed encoders for chat models. Link: https://github.com/openai/openai-cookbook/blob/main/examples...
Now English is less optimized for tokens usage and other languages are much more balanced. E.g. Ukrainian takes only twice as much tokens, before it had 6 times more tokens