News for hackers. Stuff that matters.
Re:Show HN: Franc – Detect natural languages by breuderink on Sunday October 05, 2014 @02:12PM (#8412924)
One method that I have used in the past was über-simple, yet extremely effective. It exploits ZIP compression, based on the the insight/assumption that two concatenated texts compress beter when they share their language.

I think I found it in this paper [1]. The implementation was like 13 lines of Python code. I wonder how it would compare.

[1] http://www.ccs.neu.edu/home/jaa/CSG399.05F/Topics/Papers/Ben...

Re:Show HN: Franc – Detect natural languages by wooorm on Sunday October 05, 2014 @02:40PM (#8413020)
It’s a very interesting idea. Would it work accurate enough when scaled to 160+ languages?
Re:Show HN: Franc – Detect natural languages by breuderink on Sunday October 05, 2014 @03:33PM (#8413195)
I don't know, I think I used about 40 languages. The beauty is that zip-compression captures rich statistical properties of the languages, so representation-wise it should come a long way. But counting compressed output length discretises the lang-lang distance. For shorter text this might be troubling, since this could easily result in ties. So, maybe. Perhaps I should try :).
Re:Show HN: Franc – Detect natural languages by wooorm on Monday October 06, 2014 @01:54AM (#8414458)
Perhaps you should ;) If, I’d be interest to know how it goes!