News for hackers. Stuff that matters.

A comment on Lessons from Building a Translator App That Beats Google Translate and DeepL. Parent comment

Re:Lessons from Building a Translator App That Beats Google ... by woodson on Saturday May 03, 2025 @11:25PM (#43884238)
Not sure if you tried that already, but ctranslate2 can run BART and MarianNMT models quite efficiently, also without GPUs.
[ Parent | on HN ]
Re:Lessons from Building a Translator App That Beats Google ... by omneity on Sunday May 04, 2025 @05:40AM (#43885655)
It does! I do use CT2!

On a decent CPU I found the translation to take anywhere between 15-30 seconds depending on the sentence’s length, very unnerving to me as a user.

But it’s definitely worth revisiting that. Thanks!

[ Parent | on HN ]
Re:Lessons from Building a Translator App That Beats Google ... by woodson on Sunday May 04, 2025 @05:47PM (#43889858)
Oh, that’s pretty slow. Have you tried using quantization (int8 or int8_float32)? In my experience that can help speed up CT2 execution.

Personally, I haven’t had much luck with small-ish decoder-only models (i.e., typical LLMs) for translation. Sure, GPT4 etc. work extremely well, but not so much local models capable of running on small form-factor devices. Perhaps I should revisit that.

[ Parent | on HN ]