News for hackers. Stuff that matters.
Re:Firefox is getting language translation by mantap on Tuesday October 22, 2019 @07:59AM (#21322628)
It looks useful but the lack of non-European languages makes me slightly suspicious, I wonder if their approach generalises to Arabic, Chinese, Japanese, etc.
Re:Firefox is getting language translation by yorwba on Tuesday October 22, 2019 @08:49AM (#21322988)
It depends on which part of their approch you focus on. I'd expect their machine translation model to be sufficiently general to support basically any language given enough training data. Yes, the languages you listed have some edge cases, but so do languages they already support fine. For example, the lack of spaces separating words in Chinese and Japanese can be handled by the same word-piece segmentation they need for German compound words.

The bigger problem is likely to be lack of training data. Unless they have a pile of cash to pay professional translators to produce a parallel corpus, the alternative is to scrape translations from the internet. Basically, crawl the same site multiple times with different Accept-Language headers and try to align the results. Crucially, this depends on an existing ecosystem of bilingual websites with high-quality human translations.

According to DeepL's website they're a spin-off of Linguee, who provide a search service for exactly that kind of parallel data. So before DeepL starts supporting any given language pair, you should expect it to appear in Linguee first. https://en.wikipedia.org/wiki/Linguee

Edit: It took me a while to figure out how to select a different language on https://linguee.com (Their UI seems broken using mobile Firefox Preview.) Appending /english-chinese and /english-japanese to the URL shows that they already support those two, and the alignment of translations appears reasonable to me. No /english-arabic, though.

Re:Firefox is getting language translation by azepoi on Tuesday October 22, 2019 @09:42AM (#21323479)
The European Union, the Swiss confederation, Belgium, Canada and other multilingual states, the European patent office and many international organization provide a huge corpus of professionally translated documents and reports for major European languages. Not so much for Japanese.
Re:Firefox is getting language translation by needle0 on Tuesday October 22, 2019 @08:06AM (#21322680)
Perhaps I'm too jaded, but it comes off to me as copping out of the truly difficult ones!
Re:Firefox is getting language translation by eyegor on Tuesday October 22, 2019 @11:44AM (#21324967)
If you know of a good source for large piles of docs that have been accurately/naturally translated eng/ch/jp, I'm sure deepl would be interested. As another poster pointed out, any deep learning NLP project boils down to quantity+quality of data. I'm assuming adversarial approaches don't work well in this context but I'm not very familiar with nlp research.
Re:Firefox is getting language translation by mantap on Wednesday October 23, 2019 @12:14AM (#21330973)
Right but that's kind of the essence of machine translation. You won't always have high quality parallel training data for all languages so you have to find a way to thrive with low quality data.

Google has clearly made it their mission to solve that problem, and I'd say they've been rather successful.