News for hackers. Stuff that matters.

A comment on A look at Apple’s new Transformer-powered predictive text model. Parent comment

Re:A look at Apple’s new Transformer-powered predictive text... by tgv on Sunday September 17, 2023 @02:57AM (#37542247)
Modern language processing relies on pretty large corpora, and to bootstrap it requires a part to be of high quality and annotated, although for spelling correction you don't need annotation. Or you can go the GPT-3 route and use really huge corpora. If that doesn't exist for your language, and for many smaller languages it doesn't, it won't get better than 2007.
[ Parent | on HN ]
Re:A look at Apple’s new Transformer-powered predictive text... by fauigerzigerk on Sunday September 17, 2023 @03:16AM (#37542360)
I'm not sure about that. It doesn't require a huge corpus to learn the structural features of a language. If you have that plus an English dictionary for the language, an English language model should transfer pretty well to any language.
[ Parent | on HN ]
Re:A look at Apple’s new Transformer-powered predictive text... by corbezzoli on Sunday September 17, 2023 @03:06AM (#37542298)
Indonesian is spoken by 200M people (45M of which as first language), I bet they could find some text for it. When joined with the Malaysian family, it brings the total speakers to 290M.
[ Parent | on HN ]
Re:A look at Apple’s new Transformer-powered predictive text... by bobthepanda on Sunday September 17, 2023 @04:05AM (#37542664)
My understanding is that the languages with better support have the advantage of official bodies that have section-by-section translations of all their documents; the EU into its 24 official languages and the UN with its six, for starters.
[ Parent | on HN ]
Re:A look at Apple’s new Transformer-powered predictive text... by tgv on Sunday September 17, 2023 @06:42AM (#37543578)
Building corpora has been an ongoing research activity since the 1990s. Many countries lacked the academic programs and support to build them. Often, the material isn't even there. E.g., wikipedia is a great source, but many "national" wikis are rather empty. The Bahasa wiki has 1/10th of the articles of the English one. They also seem to be shorter.

But also, Apple doesn't do much research. MS, however, has always had an active NLP research and support program, if only because of the importance of Word. But Apple depends on the research available from third parties.

[ Parent | on HN ]