News for hackers. Stuff that matters.
Re:Language models cost 10x more in some languages than others by H8crilA on Thursday May 18, 2023 @03:36AM (#35985041)
English is neither particularly efficient nor particularly inefficient. OpenAI has simply purposefully modified the distribution of tokens (deviated it from the natural distribution of all written texts) and as a result non-English text, especially non-Latin, is more expensive than it needs to be. This will disappear over time due to competitive pressure. Rapid development and fast launching leaves such wrinkles to be ironed out later.

There are no comments on this comment yet.