Re:Show HN: Franc – Detect natural languages
by wooorm on Sunday October 05, 2014 @02:40PM
(#8413020)
It’s a very interesting idea. Would it work accurate enough when scaled to 160+ languages?
Re:Show HN: Franc – Detect natural languages
by breuderink on Sunday October 05, 2014 @03:33PM
(#8413195)
I don't know, I think I used about 40 languages. The beauty is that zip-compression captures rich statistical properties of the languages, so representation-wise it should come a long way. But counting compressed output length discretises the lang-lang distance. For shorter text this might be troubling, since this could easily result in ties. So, maybe. Perhaps I should try :).
I think I found it in this paper [1]. The implementation was like 13 lines of Python code. I wonder how it would compare.
[1] http://www.ccs.neu.edu/home/jaa/CSG399.05F/Topics/Papers/Ben...