News for hackers. Stuff that matters.
Re:Show HN: Franc – Detect natural languages by wooorm on Friday October 03, 2014 @08:36AM (#8405209)
That’s because Haitians always say that! No, joking, it’s just that because of so may supported languages, the accuracy for very short inputs is extremely low.
Re:Show HN: Franc – Detect natural languages by allan_s on Friday October 03, 2014 @08:44AM (#8405240)
for that, the way I've done for TatoDetect (which is meant specifically for the task of detecting the language for "one sentence a time" ) is to have a database of N-gram huge enough for a language to be nearly sure to have "them all", so that you can consider that if your text to detect contains a N-gram that your language does not have in database, you can apply a 'decrease score' for the said language.
Re:Show HN: Franc – Detect natural languages by ppod on Friday October 03, 2014 @08:39AM (#8405219)
A regularized prior would help.
Re:Show HN: Franc – Detect natural languages by wooorm on Friday October 03, 2014 @08:44AM (#8405238)
I’m also really interested in trying something like this: http://www.slideshare.net/shuyo/short-text-language-detectio... (slide 6). But I’d need a lot of training data, more than UDHR.