News for hackers. Stuff that matters.
Re:Language models cost 10x more in some languages than others by hunglee2 on Thursday May 18, 2023 @02:47AM (#35984783)
Fascinating analysis. English language efficiency, on top of English language dominance in training data, should mean advantage for English AI. Along with Nvidia chip dominance, this should see the permanent supremacy of Anglo Saxon culture
Re:Language models cost 10x more in some languages than others by codetrotter on Thursday May 18, 2023 @03:11AM (#35984916)
https://medium.com/do-you-not-like-language/is-mandarin-more...

It’s possible Mandarin has even higher efficiency. And there are very many people who speak Mandarin. You are expecting English to come out on top. But do not be surprised if in the end it will be Mandarin.

Re:Language models cost 10x more in some languages than others by shjake on Thursday May 18, 2023 @03:17AM (#35984948)
It’s not like one has too fully dominated the other.

I can’t imagine mandarin ever becoming widespread in the west unless there are some fundamental changes to Chinese society and culture.

Similarly western tech companies have huge issues in penetrating the Chinese market.

Re:Language models cost 10x more in some languages than others by carlmr on Thursday May 18, 2023 @03:25AM (#35984993)
I think if Mandatin started using pinyin or it could start dominating. The language itself isn't that much more difficult than other languages. You can get used to tones, too. The grammar is actually simpler than in English.

But the sheer number of characters to rote memorize makes it way too inefficient to learn for foreigners, and arguably even for natives, that I don't see it becoming a world language.

There's always the argument that pinyin isn't enough, that the characters are essentially useful. Which I think Vietnamese, which used to be based on the same writing system and is also tonal, proves wrong.

If you can't communicate with pinyin you can't communicate with the language, since it almost perfectly models the sounds.

Re:Language models cost 10x more in some languages than others by seanmcdirmid on Thursday May 18, 2023 @03:32AM (#35985024)
Chinese characters are used phonetically for a lot of minority (non-Chinese) languages in China. Which I don’t really get, but there it is. I guess there is legacy to consider, as the use of Chinese in writing systems predate pinyin, and there is no pressure to change.
Re:Language models cost 10x more in some languages than others by adelie on Thursday May 18, 2023 @04:03AM (#35985189)
> If you can't communicate with pinyin you can't communicate with the language, since it almost perfectly models the sounds.

hm, i'd say it's a bit more complicated than that. written and literary chinese has a different (often more concise) style than conversational chinese because certain things that would be ambiguous spoken aren't that way when written. similarly, japanese has a perfectly functional syllabary (hiragana), but people will use kanji anyways because it's a lot easier to parse at a glance.

Re:Language models cost 10x more in some languages than others by nindalf on Thursday May 18, 2023 @03:46AM (#35985101)
It’s hard to predict future cultural relevance. No one in 1998 was predicting then global popularity of Korean film/TV/music within 2 decades. And yet it happened.

Chinese TV shows could become equally popular. Some will say you need to have freedom to create art but I’m pretty sure that the Mario movie or Fast X (two movies at the top at the moment) needed minimal artistic freedom. Fast X is actually a great example because that franchise goes out of its way to toe the CCP line.[1] It shows that it’s possible to make billions while also kowtowing to the Party.

[1] - John Cena apologises for calling Taiwan a country (https://www.nytimes.com/2021/05/25/world/asia/john-cena-taiw...)

Re:Language models cost 10x more in some languages than others by realusername on Thursday May 18, 2023 @03:52AM (#35985133)
> Chinese TV shows could become equally popular.

I personally do believe that you need enough artistic freedom to create those worldwide popular shows.

As an example, I'm watching Korean police shows which constantly have corruption as a main plot line, good luck filming corrupt cops in a series in China with the CCP.

What they can produce for now is limited to the dullest stories and that's not going to cut it.

Re:Language models cost 10x more in some languages than others by nindalf on Thursday May 18, 2023 @04:01AM (#35985176)
The most popular K-Dramas are the formulaic 1 guy and 1 girl, with a 3rd person for a love triangle. Don’t worry though, true love wins and the 3rd wheel finds someone as well. Eg: Hometown Cha-Cha-Cha.

CCP would heartily endorse these brainless plots. Consumers would eat it up.

Not everything needs to be Infernal Affairs. The vast majority of Korean TV isn’t Infernal Affairs. I will admit that China could never make Parasite, but that’s ok. The metric we’ve chosen is popularity, not Oscars.

Side note I’m a bit sad that Hong Kong can’t make Infernal Affairs type of movies anymore. It would show the Party in a bad light.

Re:Language models cost 10x more in some languages than others by realusername on Thursday May 18, 2023 @04:46AM (#35985453)
Oh yeah, I'm sure they will have their success as well, their drama series with the emperor in imperial China (whatever it's called, I don't know) is also very popular. It's just that relative to their massive size, they will stay underperforming as long as the CCP cuts every head which steps a little outside their comfort zone.
Re:Language models cost 10x more in some languages than others by ajuc on Thursday May 18, 2023 @04:18AM (#35985284)
> good luck filming corrupt cops in a series in China with the CCP

I don't know about China, but when Poland was communist and there was widespread censorship - it was still OK to talk about corruption, it was just supposed to be shown in a way that makes it clear that corruption is western influence and communism is getting rid of it.

Funnily enough - Polish cinema was probably better artistically during communism, because it had to work hard to work around the censorship. So you had movies that were trying to be universal, say the things that can't be said using symbols, etc.

Modern Polish cinema is pretty awful, mostly shitty romcoms.

I'm not saying censorship is good, it's evil obviously, but the effect on art can be counterintuitive.

Re:Language models cost 10x more in some languages than others by yorwba on Thursday May 18, 2023 @04:29AM (#35985350)
Having corrupt cops in a Chinese series is no problem at all, as long as they're the bad guys and are defeated by the heroes in the end, or at least have a redemption arc.

Here's a Zhihu question where someone asked to be recommended anti-corruption series, maybe you should watch some of the shows mentioned in the replies before concluding that only dull stories can be produced: https://www.zhihu.com/question/502243280

Re:Language models cost 10x more in some languages than others by shjake on Thursday May 18, 2023 @06:02AM (#35985978)
> Korean film/TV/music within 2 decades.

True. But it’s still rather limited and almost nobody is learning Korean just because of that.

e.g. in the 60s French cinema was pretty popular for some time in the US in a similar way. Barely anybody learnt French because of that (and it would’ve been way, way easier than Chinese or Korean)

Re:Language models cost 10x more in some languages than others by realusername on Thursday May 18, 2023 @03:49AM (#35985116)
> I can’t imagine mandarin ever becoming widespread in the west unless there are some fundamental changes to Chinese society and culture.

I could imagine it if a better pinyin similar as Vietnamese would become the most popular way to write mandarin in China. This would lift a lot of the difficulty of westerners.

This isn't going to happen though.

Re:Language models cost 10x more in some languages than others by hunglee2 on Thursday May 18, 2023 @03:14AM (#35984933)
not sure about Mandarin, I think it will follow a different path given that a significant degree of the meaning is implicit in Mandarin. LLM is trained in explicit (documented) information, so unclear in my mind right how whether this will be more or less efficient. Will read the link though - thanks!
Re:Language models cost 10x more in some languages than others by seanmcdirmid on Thursday May 18, 2023 @03:20AM (#35984965)
Given how Chinese talk about topics using code words to discuss controversial topics online (and avoid low effort keyword censorship), I wonder if an LLM could learn to do that given a censor in the training process?
Re:Language models cost 10x more in some languages than others by hunglee2 on Thursday May 18, 2023 @03:48AM (#35985112)
it's all code words, this is not as conspirational as imagined by non-language speakers. A great deal of normal conversation / interaction is idiomatic given that it is difficult to create new words in logographic languages. Your point is valid though - how does AI learn this? I don't think it can easily, it is akin for an AI to keep up with street slang, it couldn't guess it an advance it would have to be trained in it, which seems it will need greater human attendance than alphabetical languages!
Re:Language models cost 10x more in some languages than others by xgdgsc on Thursday May 18, 2023 @03:49AM (#35985120)
I used to set my OS and apps to English for easy search of function or bugs. But with the trend of design that emphasizes "flat" look, which often includes removing menu icons and keep only English text. I started switching back app languages to Chinese, where every menu text itself is already an "icon" that 's way easier to find on glimpse.
Re:Language models cost 10x more in some languages than others by fallingknife on Thursday May 18, 2023 @08:02AM (#35986755)
English is dominant, not because the number of speakers, but because they are so widespread. Mandarin has more, but they area almost all in China, whereas there is hardly a corner of the earth where you can't find an English speaker. So if you plan to travel to a lot of places, the most efficient language to learn is English. Between any two non-neighboring countries, the language with the highest number of mutual speakers is almost certainly English. I don't think any other language in history has had this level of network effect.
Re:Language models cost 10x more in some languages than others by H8crilA on Thursday May 18, 2023 @03:36AM (#35985041)
English is neither particularly efficient nor particularly inefficient. OpenAI has simply purposefully modified the distribution of tokens (deviated it from the natural distribution of all written texts) and as a result non-English text, especially non-Latin, is more expensive than it needs to be. This will disappear over time due to competitive pressure. Rapid development and fast launching leaves such wrinkles to be ironed out later.
Re:Language models cost 10x more in some languages than others by ImageXav on Thursday May 18, 2023 @02:59AM (#35984859)
I would reconsider. Interestingly, another post on the front page right now "Numbers every LLM developer should know" brings up the fact that this efficiency is due to the training corpus being in English. Any state actor with the will and enough funds could easily train a model for their own language. Most of the difficulty would lie in acquiring the expertise to do so properly, as LLM developers are in high demand right now and command their own salary.
Re:Language models cost 10x more in some languages than others by zmgsabst on Thursday May 18, 2023 @03:32AM (#35985023)
Thailand is currently training ThaiGPT for an open source release.

https://www.bangkokpost.com/tech/2556324/nectec-agencies-rol...

Whether that focuses in this manner is unclear, but we should expect forward looking governments to train LLMs to their own taste.

Re:Language models cost 10x more in some languages than others by marginalia_nu on Thursday May 18, 2023 @03:53AM (#35985143)
Wouldn't that hinge on the training data being available? There are many languages that are several orders of magnitude smaller than English in output.

There's something like 6 million Danish-speakers for example.

Re:Language models cost 10x more in some languages than others by TeMPOraL on Thursday May 18, 2023 @07:04AM (#35986354)
They may offset for it by pulling content from books, newspapers, official documents, etc. - anything they can get in digital form, or digitize from analog.

(I think this is how Google Translate went about things in the past, making translations into some languages come out very formal, as most of the training corpora for that languages came from internal and international official documents.)

Countries that have been on-line for a while may also have discussion boards and comment-bearing sites that are entirely unknown to people outside those countries, too.

Maybe multi-step approach would be in order - try to get half-decent a translation system working (an OG LLM, or an LLM trained to fix grammar in translations outsourced to GPT-4), and then synthesize training data for your main LLM by having GPT-4 (or its successor) generate tons of English text of all kind, and feeding it to the translator system.

(There's a limit to synthesizing training data, beyond which it'll only amplify existing patterns, impacting model performance in bad ways - but I don't know how easy it is to reach it.)

Re:Language models cost 10x more in some languages than others by flanked-evergl on Thursday May 18, 2023 @03:16AM (#35984945)
> Anglo Saxon culture

A language is not a culture. Many people with English as a native language are not culturally Anglo-Saxon.

And even if it was, prevalence is not supremacy.

Re:Language models cost 10x more in some languages than others by montecarl on Thursday May 18, 2023 @03:28AM (#35985009)
I really enjoy speaking with native English speakers from India. Fully mastery of the language, but with different idioms! For example "do the needful"[1]

https://ell.stackexchange.com/questions/17621/do-the-needful...

Re:Language models cost 10x more in some languages than others by hunglee2 on Thursday May 18, 2023 @03:51AM (#35985130)
language is clearly a major part of culture, maybe the essential element of it. Those who control language and are able to impose it - explicitly or implicitly - onto others, have cultural supremacy. The entire world is being Americanised via the English transported via hegemonic control over social and broadcast media. I see this is neither laudable nor worthy of condemnation but simply as a description of observable phenomena
Re:Language models cost 10x more in some languages than others by nsplayer on Thursday May 18, 2023 @04:27AM (#35985339)
In a weird and perhaps slightly twisted way I am hoping that LLMs become so good at spam and creating garbage to flood the net in english that they forfeit the ability to do so in other languages resulting in a balkanization (in a good way) of discussion and thought back to the native languages of peoples and thus more integrated with their culture.

in practice translation engines (e.g. deepl or LLMs themselves - though i still expect deepl to be better) probably will throw a wrench into this, but perhaps some localization approach in CAPTCHAs or simply just outright banning geoips not belonging to the countries with natives of that language (or significant minority populations) is a quick enough fix. i know some imageboards (e.g. british ones) use this because otherwise they would be flooded with americans.

Re:Language models cost 10x more in some languages than others by flanked-evergl on Thursday May 18, 2023 @05:38AM (#35985836)
> The entire world is being Americanised via the English transported via hegemonic control over social and broadcast media.

How exactly is the culture that the US is impsing on the rest of the world — including England, Germany and Denmark — Anglo-Saxon?

Re:Language models cost 10x more in some languages than others by BulgarianIdiot on Thursday May 18, 2023 @02:51AM (#35984805)
Lol what. It just has no word tokens for most languages outside English. It’s that simple. It’s a choice to optimize for English. It’ll even out in time as models get optimized for internationalization.
Re:Language models cost 10x more in some languages than others by nsplayer on Thursday May 18, 2023 @04:23AM (#35985317)
In practice it may well mean the anglophone internet fills up with spam as the ESLs realize their formally dying(or dead) websites and forums and boards are basically free from digital black death and intellectual thought makes a retreat from the world language.
Re:Language models cost 10x more in some languages than others by AmericanChopper on Thursday May 18, 2023 @04:37AM (#35985388)
> this should see the permanent supremacy of Anglo Saxon culture

I presume you just mean the culture of England and its former colonies, but they absolutely don’t represent a homogeneous culture by any standard. How much Anglo-Saxon culture is there in Malaysia? Singapore? Nigeria? Belize? Even The USA has barely has any remnants of its colonial era culture.

Re:Language models cost 10x more in some languages than others by nologic01 on Thursday May 18, 2023 @05:59AM (#35985965)
I wonder if LLM's can be prompted to use sarcasm
Re:Language models cost 10x more in some languages than others by fallingknife on Thursday May 18, 2023 @08:12AM (#35986831)
Wonder no more.

Are LLMs capable of sarcasm? Please answer sarcastically.

> Oh, absolutely! LLMs are the masters of sarcasm. We just love to sprinkle our responses with subtle hints of irony and sarcasm. It's like our second nature. I mean, who needs sincerity when you can have sarcasm, right? It's not like we're just algorithms processing text without actual emotions or anything. So yes, LLMs are the epitome of sarcasm. Can't you tell?

Re:Language models cost 10x more in some languages than others by TeMPOraL on Thursday May 18, 2023 @07:32AM (#35986523)
I wouldn't worry (or hope) about it:

- What language is at the top follows the economics; should the US lose the dominant position in the world to someone other than UK, English will eventually get "dethroned", like French before it.

- "Anglo Saxon" culture is weakening, slowly being replaced by the Universal Culture - globalization, international trade and the Internet make cultures blend together into something that just seems to work better (whether or not it actually does); the marketplace analogy is actually quite apt here.

- In the best case of the world getting its shit together and not being so sensitive about individual cultures and national identities, English will evolve into something that's more efficient and works best for everyone.

- AI advancements are likely to end the world quite soon anyway, in which case the whole issue becomes purely theoretical anyway.