Re:Watson Services
by pesenti on Wednesday October 08, 2014 @08:06AM
(#8426439)
Sorry Polish is not one of the supported language yet: Arabic; Chinese (Simplified); Chinese (Traditional); Cyrillic; Danish; Dutch; English; Farsi; Finnish; French; German; Greek; Hebrew; Hindi; Icelandic; Italian; Japanese; Korean; Norwegian (Bokmal); Norwegian (Nynorsk); Portuguese; Spanish; Swedish; Turkish; Urdu.
[ Parent
| on HN ]
Re:Watson Services
by angrypoulpe on Wednesday October 08, 2014 @10:03AM
(#8426885)
French is identified as EN_us -> http://imgur.com/VdDJVw4 Inadmissible !
AYLIEN's Text API does a better job :-) http://aylien.com/text-api-demo?text=vache%20cochon%20metal%...
(disclaimer: I'm the founder of AYLIEN)
I also imagine that distinguishing between many of these languages is not a trivial task for a computer to handle.
You'd get collisions if you're trying to determine the language of a single word, obviously. But a sentence or two should be no problem. You're just matching words in a dictionary.
Different languages are mostly quite distinct. It would be a more interesting challenge to recognize and understand, say, German dialects.
In fact you can already get very good results, even without a dictionary, just with a distribution of the trigrams in your input languages (which is immediate to compute from a corpus). Just compute the distribution of the sample and return the language that has the closest one.
Even a single letter frequency histogram is already a very powerful discriminator to detect language. Although I suppose scoring maximum likelihood on bi- or trigrams works better for very small (sub-sentence) bits of text.
Oh right, I'm sorry. It was actually written on the demo page, but I somehow missed it.
Odd that Farsi is supported, not Polish, given that Poland's GDP is close to Iran's and that Poland has no sanctions against it. Perhaps IBM is planning to sell lots of Watson-powered systems in Afghanistan & Tajikistan?
Incidentally, Poland's GDP is several times that of Finland…
Do you not think there might be a market for software for analyzing text in Farsi perhaps -outside- the region where it's actually spoken?
(disclaimer - work for IBM, but not on Watson, or anything related to intelligence analytics)
Polish as a language is more similar to other languages than Finnish or Farsi. Basically, it's easier to detect Farsi and Finnish because they're more distinct and in smaller language families than Polish, which is similar to many slavic languages.