Re:Language models cost 10x more in some languages than others
by H8crilA on Thursday May 18, 2023 @03:36AM
(#35985041)
English is neither particularly efficient nor particularly inefficient. OpenAI has simply purposefully modified the distribution of tokens (deviated it from the natural distribution of all written texts) and as a result non-English text, especially non-Latin, is more expensive than it needs to be. This will disappear over time due to competitive pressure. Rapid development and fast launching leaves such wrinkles to be ironed out later.
[ Parent
| on HN ]
There are no comments on this comment yet.