News for hackers. Stuff that matters.
Re:Why is GPT-3 15.77x more expensive for certain languages? by crazygringo on Monday April 10, 2023 @03:17PM (#35516203)
Words aren't an equivalent count between languages either. English uses a lot of helper words, some other languages use multiple suffixes. Chinese characters don't even make it clear where "word" boundaries are -- there are no spaces.
Re:Why is GPT-3 15.77x more expensive for certain languages? by mgaunard on Monday April 10, 2023 @03:22PM (#35516258)
Chinese does make it explicit where word boundaries are.

The only language that doesn't is Thai, but there are still well-documented algorithms for it.

Re:Why is GPT-3 15.77x more expensive for certain languages? by crazygringo on Monday April 10, 2023 @03:27PM (#35516331)
How does it make it explicit? You need a dictionary to figure it out, no? Same as e.g. Japanese?
Re:Why is GPT-3 15.77x more expensive for certain languages? by kccqzy on Monday April 10, 2023 @03:53PM (#35516639)
Right but such dictionaries are already built in to all major operating systems. The double-click-to-select-word interaction works well with Chinese and Japanese in all major operating systems. Without such dictionaries you can't even implement word selection.
Re:Why is GPT-3 15.77x more expensive for certain languages? by fomine3 on Monday April 10, 2023 @09:02PM (#35519364)
It works until it recognizes 外国人参政権 as foreign/carrot/regime
Re:Why is GPT-3 15.77x more expensive for certain languages? by biztos on Monday April 10, 2023 @03:49PM (#35516582)
Really, only Thai? Is there a reference for that? A quick search suggests it’s not the case, but I’m no expert.

As a lowly beginner I find the lack of word boundaries in Thai frustrating but I think it’s just that I have not yet learned to think in syllables, I’m still always sounding them out in my head until I have a word I recognize, there’s no flow.

This seems like something the LLMs should be very good at. Google Translate does OK-ish while Apple just throws up its hands in frustration and refuses to translate Thai texts.

Re:Why is GPT-3 15.77x more expensive for certain languages? by mgaunard on Tuesday April 11, 2023 @03:12AM (#35521844)
Read the Unicode standard, it covers all of these things.