It seems to work brilliantly with Russian. Less than 5 seconds to translate a medium length wikipedia article on my computer. This exceeds my hopes and expectations.

To be fair, Wikipedia basically appears in its entirety in the training data. It’s a good test to see whether the translation model and all the plumbing works well, but not whether the model generalises well.

There's plenty of articles on wikipedia where different language versions of a page are vastly different (it feels like the majority in my experience, but that's no proof of course), how would that be useful as training data unless heavily curated?

The datasets these models are trained on are sentence pairs. So even if just a couple of sentences between two wikipedia sites are direct translations of each other, they will have appeared in the training set. They don’t have to have appeared on the same topic page, it could be that English Wikipedia has a whole category for a topic while Estonian Wikipedia has just a long single page, direct translations will still be identified and used in training.

I also think that the domain and the type of language used on Wikipedia is pretty consistent which will help a lot with unseen sentences.

By no means are these models bad! It’s just that Wikipedia is a particularly easy test for them.

How are these identified? Are they human curated? If not it seems like you need a translator to decide if they are equivalent sentence pairs to build your translator.

You're pretty much right on the money. For ParaCrawl[1] (which I worked on) we used fast machine translation systems that were "good enough" to translate one side of each pair to the language of the other, see whether they'd match sufficiently, and then deal with all the false positives through various filtering methods. Other datasets I know of use multilingual sentence embeddings, like LASER[2], to compute the distance between two sentences.

Both of these methods have a bootstrapping problem, but at this point in the MT for many languages we have enough data to get started. Previous iterations of ParaCrawl used things like document structure and overlap of named entities among sentences to identify matching pairs. But this is much less robust. I don't know how they solve this problem today for low-resource languages.

[1] https://paracrawl.eu

[2] https://github.com/yannvgn/laserembeddings

My experience with occasional use over the past four months or so is that newspapers, business documents, and other formal documents often appear to have essentially perfect translation, at least in terms of sounding plausable without understanding the language translated. For more casual stuff like twitter posts or song lyrics it makes plenty of obvious errors, although of course some of that stuff makes little sense as written. It seems to have different issues than Google Translate, although not much if any worse overall from my limited experience.

It sounds like after Bergamot funding ended there were some communication issues and the Translate Locally group that was working with the Firefox Translate group stopped working together and now have their own extension, as mentioned in another comment:

https://news.ycombinator.com/item?id=33795141

That one can be installed outside the browser and would likely give better performance, although I haven't tried that yet.

The other interesting open source translation software I've seen is Apertium:

https://github.com/apertium

I don't think they have a browser extension unfortunately but it is an entirely rule based translation rather than AI models. I haven't tried this one yet either but hope to soon. I did try their web interface a few times:

https://www.apertium.org

It really is surprisingly readable. Does excellent work on german, spanish & italian f1 news articles.

I used to use Android Chrome when overseas because auto translating of web pages is indispensable. It was a major concession for me, because I loath Google preventing ad blocking on Android Chrome. Still, it was less than perfect - constant popups asking you about the quality of the translation. The delay in sending it to Google wasn't noticeable when you had a good connection, but on the edge of mobile coverage it made web browsing even more miserable.

Then this came along. All those nits are gone. Personally I find the translations easier to understand than Google's. (When you're overseas easier to understand trumps grammaticality perfect every time.) I'm not a fan of the banner at the top - the could move it to a tool bar icon like Ublock Origin does, but apart from that - it's damned good.

Now we need a replacement for Google Lens. For all it's flaws, Lens seems near magical to me.