News for hackers. Stuff that matters.

A comment on Language detection with Google's Compact Language Detector. Parent comment

Re:Language detection with Google's Compact Language Detector by ninjin on Saturday October 22, 2011 @11:12AM (#3143642)
Mark Pilgrim reversed (or ripped out, can't remember) the encoding detection that Firefox uses. It has done a fairly good job for my web crawling:

http://pypi.python.org/pypi/chardet

[ Parent | on HN ]
Re:Language detection with Google's Compact Language Detector by e98cuenc on Saturday October 22, 2011 @11:20AM (#3143655)
In my experience chardet misclassifies very often iso-8859-1 as iso-8859-2. I saw the misclassification even in small spanish pages, which were using only the typical spanish characters.
[ Parent | on HN ]