News for hackers. Stuff that matters.

A comment on Show HN: Franc – Detect natural languages. Parent comment

Re:Show HN: Franc – Detect natural languages by hywel on Friday October 03, 2014 @08:29AM (#8405184)
Based on a 2-sec look at the code, it's using a built-in database of trigrams as a predictor of the language.

https://github.com/wooorm/franc/blob/master/lib/data.json

[ Parent | on HN ]
Re:Show HN: Franc – Detect natural languages by allan_s on Friday October 03, 2014 @08:32AM (#8405192)
my bad, I've been looking to data folder first and haven’t found anything, I should have tried harder
[ Parent | on HN ]
Re:Show HN: Franc – Detect natural languages by riffraff on Friday October 03, 2014 @10:39AM (#8405710)
the question would be where he got the language data

If the original language data is available I'd suggest classifying the trigrams as "high" and "low" frequency, which should improve performance without needing to keep full frequency data.

[ Parent | on HN ]
Re:Show HN: Franc – Detect natural languages by wooorm on Friday October 03, 2014 @10:44AM (#8405731)
No full-frequency data is kept, only 300 top-trigrams are identified. A quick through the source also reveals wooorm/trigrams, and wooorm/udhr, as sources!
[ Parent | on HN ]
Re:Show HN: Franc – Detect natural languages by riffraff on Friday October 03, 2014 @01:37PM (#8406556)
yes, I meant: keeping full frequency could have been avoided to save space/memory but having two classes high/low could be a good tradeoff.
[ Parent | on HN ]
Re:Show HN: Franc – Detect natural languages by wooorm on Friday October 03, 2014 @03:42PM (#8407263)
It’s an interesting thought. I might fiddle on it, but I’m not sure it would work in practice (d’oh). Thanks!
[ Parent | on HN ]