News for hackers. Stuff that matters.

A comment on Show HN: My Leeds Hack Day project, a language detection API. Parent comment

Re:Show HN: My Leeds Hack Day project, a language detection API by lgeek on Tuesday November 09, 2010 @01:55PM (#1887487)
That's pretty cool! One of my friends suggested to use dictionaries and bloom filters but I've wanted to build probabilistic language models.

Do you have any accuracy stats? I'm guessing my approach might work better in some cases because the models include frequency information too. Did you experience significant accuracy loss when adding new languages? Anyway, I'll run it over my test data and compare.

[ Parent | on HN ]
Re:Show HN: My Leeds Hack Day project, a language detection API by petercooper on Tuesday November 09, 2010 @03:28PM (#1887785)
Do you have any accuracy stats?

No, but as you have noted, the method has the intrinsic property of being less accurate with fewer words and more accurate the longer the text. As my anticipated use was for documents over 10-20 words, this was OK. I expect the other techniques I outlined that I'm switching to to yield more accurate results across the board.

[ Parent | on HN ]
Re:Show HN: My Leeds Hack Day project, a language detection API by lgeek on Tuesday November 09, 2010 @03:13PM (#1887740)
...and these are my results: http://imgur.com/9U6II.png I've only kept English, French and German for WhatLanguage. It looks like the Markov chains approach is indeed more accurate for shorter phrases but the dictionary approach gets slightly better for longer sentences.
[ Parent | on HN ]