Re:Displaying Japanese and English Text on the Web
by kevin_thibedeau on Friday February 08, 2019 @05:53PM
(#19119452)
I've been scraping some Japanese sites lately and this has been a minor annoyance. Content-Type rarely has the encoding and Requests doesn't default to UTF-8 so you get mojibake for EUC-JP and UTF-8 unless you intervene.
In HTML5, the browser is supposed to not sniff the document for a meta charset if the server headers specify a charset in the Content-type.