News for hackers. Stuff that matters.
Re:Watson Services by johnward on Wednesday October 08, 2014 @08:19AM (#8426486)
I also imagine that distinguishing between many of these languages is not a trivial task for a computer to handle.
Re:Watson Services by TillE on Wednesday October 08, 2014 @08:47AM (#8426587)
You'd get collisions if you're trying to determine the language of a single word, obviously. But a sentence or two should be no problem. You're just matching words in a dictionary.

Different languages are mostly quite distinct. It would be a more interesting challenge to recognize and understand, say, German dialects.

Re:Watson Services by a3_nm on Wednesday October 08, 2014 @09:08AM (#8426659)
In fact you can already get very good results, even without a dictionary, just with a distribution of the trigrams in your input languages (which is immediate to compute from a corpus). Just compute the distribution of the sample and return the language that has the closest one.
Re:Watson Services by tripzilch on Saturday October 11, 2014 @04:36AM (#8441635)
Even a single letter frequency histogram is already a very powerful discriminator to detect language. Although I suppose scoring maximum likelihood on bi- or trigrams works better for very small (sub-sentence) bits of text.