Re:Why is GPT-3 15.77x more expensive for certain languages?
by crazygringo on Monday April 10, 2023 @03:17PM
(#35516203)
Words aren't an equivalent count between languages either. English uses a lot of helper words, some other languages use multiple suffixes. Chinese characters don't even make it clear where "word" boundaries are -- there are no spaces.
[ Parent
| on HN ]
Re:Why is GPT-3 15.77x more expensive for certain languages?
by mgaunard on Monday April 10, 2023 @03:22PM
(#35516258)
Chinese does make it explicit where word boundaries are.
The only language that doesn't is Thai, but there are still well-documented algorithms for it.
Re:Why is GPT-3 15.77x more expensive for certain languages?
by crazygringo on Monday April 10, 2023 @03:27PM
(#35516331)
How does it make it explicit? You need a dictionary to figure it out, no? Same as e.g. Japanese?
Re:Why is GPT-3 15.77x more expensive for certain languages?
by kccqzy on Monday April 10, 2023 @03:53PM
(#35516639)
Right but such dictionaries are already built in to all major operating systems. The double-click-to-select-word interaction works well with Chinese and Japanese in all major operating systems. Without such dictionaries you can't even implement word selection.
Re:Why is GPT-3 15.77x more expensive for certain languages?
by fomine3 on Monday April 10, 2023 @09:02PM
(#35519364)
It works until it recognizes 外国人参政権 as foreign/carrot/regime
Re:Why is GPT-3 15.77x more expensive for certain languages?
by biztos on Monday April 10, 2023 @03:49PM
(#35516582)
Really, only Thai? Is there a reference for that? A quick search suggests it’s not the case, but I’m no expert.
As a lowly beginner I find the lack of word boundaries in Thai frustrating but I think it’s just that I have not yet learned to think in syllables, I’m still always sounding them out in my head until I have a word I recognize, there’s no flow.
This seems like something the LLMs should be very good at. Google Translate does OK-ish while Apple just throws up its hands in frustration and refuses to translate Thai texts.