News for hackers. Stuff that matters.

A comment on Llama 4 performs worse than Llama 3 at translation.

Re:Llama 4 performs worse than Llama 3 at translation by guccihat on Sunday April 06, 2025 @07:06AM (#43600546)
The article concludes that the overall translation score of Llama 4 is below that of Llama 3.3. However, the included table shows that Llama 4 scores better on all subcategories included in the test - coherence, idiomaticity and accuracy.

Something does not add up. The conclusion just states "...downgrade from LLama 3.3 in every respect" without further explanation.

[ Parent | on HN ]
Re:Llama 4 performs worse than Llama 3 at translation by smallerize on Sunday April 06, 2025 @07:25AM (#43600632)
Looking at the individual language pages, it does come behind pretty often. And in Japanese for example, it has higher scores but also a much higher refusal rate. The summary page doesn't show a refusal rate column, so not all the data is represented there.
[ Parent | on HN ]
Re:Llama 4 performs worse than Llama 3 at translation by Tomte on Sunday April 06, 2025 @07:21AM (#43600611)
Could be Simpson's Paradox: https://en.wikipedia.org/wiki/Simpson%27s_paradox
[ Parent | on HN ]