Language detection with Google's Compact Language Detector

Re:Language detection with Google's Compact Language Detector by sick-boy on Saturday October 22, 2011 @07:28AM (#3143340)

"You must provide it clean (interchange-valid) UTF-8, so any encoding issues must be sorted out before-hand."

In most cases you have to know the language in order to guess the encoding and convert to UTF-8 if necessary. Mutual recursion...

[ Parent | on HN ]

Re:Language detection with Google's Compact Language Detector by adambyrtek on Saturday October 22, 2011 @08:51AM (#3143442)

Encoding, but obviously not language, should be provided explicitly as metadata (e.g. Content-Type HTTP header). Also, most of content available on the web is already UTF-8 (65.9% according to a recent survey[1]).

[1] http://w3techs.com/technologies/details/en-utf8/all/all

[ Parent | on HN ]

Re:Language detection with Google's Compact Language Detector by ninjin on Saturday October 22, 2011 @11:12AM (#3143642)

Mark Pilgrim reversed (or ripped out, can't remember) the encoding detection that Firefox uses. It has done a fairly good job for my web crawling:

http://pypi.python.org/pypi/chardet

[ Parent | on HN ]

Re:Language detection with Google's Compact Language Detector by e98cuenc on Saturday October 22, 2011 @11:20AM (#3143655)

In my experience chardet misclassifies very often iso-8859-1 as iso-8859-2. I saw the misclassification even in small spanish pages, which were using only the typical spanish characters.

[ Parent | on HN ]

Re:Language detection with Google's Compact Language Detector by Maakuth on Saturday October 22, 2011 @07:36AM (#3143356)

I'd say in most cases UTF-8 is already used. Of course this depends on the source of the text.

[ Parent | on HN ]