News for hackers. Stuff that matters.
Re:The Nordic Token Tax by vpaulus on Tuesday November 03, 2026 @01:42AM (#50348820)
Finnish is in a totally different language family than Swedish-Norwegian-Danish. However Viking performs the same result. I am wondering the outcome when using Viking for Hungarian, as it is a near relative to Finnish.
Re:The Nordic Token Tax by silfverstrom on Tuesday November 03, 2026 @03:02AM (#50349312)
Viking was trained on Finnish as well, so that's why it gets good results. I ran a quick check on Hungarian, Viking didn't perform well at all(about 2x the tokens of English).
Re:The Nordic Token Tax by ahartmetz on Tuesday November 03, 2026 @06:19AM (#50350591)
Would have been interesting to include German and Dutch. The German training corpus must be much bigger while the language is roughly the same "distance" from English as other Germanic languages except Dutch. Dutch is most similar to English, but likely has a smaller training corpus than German.
Re:The Nordic Token Tax by Frieren on Tuesday November 03, 2026 @08:21AM (#50351657)
Very interesting, now I really wonder about Spanish, and Mandarin Chinese. There is so much written literature in Spanish that I would guess that much of it has been used to train models. For Chinese is a similar case but it has the interesting side effect to be a very different language than English. (English has germanic roots, but most of its words are latin based so not so far away from Spanish).