News for hackers. Stuff that matters.

A comment on Show HN: Franc – Detect natural languages. Parent comment

Re:Show HN: Franc – Detect natural languages by riffraff on Friday October 03, 2014 @10:39AM (#8405710)
the question would be where he got the language data

If the original language data is available I'd suggest classifying the trigrams as "high" and "low" frequency, which should improve performance without needing to keep full frequency data.

[ Parent | on HN ]
Re:Show HN: Franc – Detect natural languages by wooorm on Friday October 03, 2014 @10:44AM (#8405731)
No full-frequency data is kept, only 300 top-trigrams are identified. A quick through the source also reveals wooorm/trigrams, and wooorm/udhr, as sources!
[ Parent | on HN ]
Re:Show HN: Franc – Detect natural languages by riffraff on Friday October 03, 2014 @01:37PM (#8406556)
yes, I meant: keeping full frequency could have been avoided to save space/memory but having two classes high/low could be a good tradeoff.
[ Parent | on HN ]
Re:Show HN: Franc – Detect natural languages by wooorm on Friday October 03, 2014 @03:42PM (#8407263)
It’s an interesting thought. I might fiddle on it, but I’m not sure it would work in practice (d’oh). Thanks!
[ Parent | on HN ]