Re:Code Page 437 Refuses to Die
by masklinn on Sunday May 08, 2016 @11:44AM
(#11654479)
> let me rephrase: it is, but it doesn't have to be. C family is a nightmare, Python et al are delightful
Unless you do multiplatform development, then the language has a hard time saving you (and Python definitely does not)
> (Another irony: with UTF8 gaining more and more mind share, encoding issues actually become harder to find and debug: they don't show up, and when they do, fewer and fewer people know how to deal with them. Everyone switching to UTF8 just hides the bugs, until it doesn't.)
That's not true at all. A ton of byte sequences (and 13 standalone bytes) are outright illegal in UTF8, there's a fair amount of error handling in a validating UTF8 decoder[0], whereas there usually isn't any invalid byte (let alone byte sequence) in 8-bit codepages or character sets. When you decode random bytes in Windows-1256 or ISO-8859-9[1] and re-encode them as UTF8, the UTF8 encoder isn't the one at fault for your garbage output, as far as I could see it got perfectly valid unicode data.
People passing through unvalidated data (possibly assuming it's UTF8) isn't a problem with UTF8 either, by the way.
[0] whether that's used in strict mode or in replacement mode is a different concern and not a blemish on UTF8 itself
[1] which will always succeed, you can try it at home, just get a bunch of bytes from urandom and feed them to various decoders, chances are low that you'll generate anything the UTF8 decoder will accept, chances are also low that you'll generate anything an ISO-8859 character set will reject.
Unless you do multiplatform development, then the language has a hard time saving you (and Python definitely does not)
> (Another irony: with UTF8 gaining more and more mind share, encoding issues actually become harder to find and debug: they don't show up, and when they do, fewer and fewer people know how to deal with them. Everyone switching to UTF8 just hides the bugs, until it doesn't.)
That's not true at all. A ton of byte sequences (and 13 standalone bytes) are outright illegal in UTF8, there's a fair amount of error handling in a validating UTF8 decoder[0], whereas there usually isn't any invalid byte (let alone byte sequence) in 8-bit codepages or character sets. When you decode random bytes in Windows-1256 or ISO-8859-9[1] and re-encode them as UTF8, the UTF8 encoder isn't the one at fault for your garbage output, as far as I could see it got perfectly valid unicode data.
People passing through unvalidated data (possibly assuming it's UTF8) isn't a problem with UTF8 either, by the way.
[0] whether that's used in strict mode or in replacement mode is a different concern and not a blemish on UTF8 itself
[1] which will always succeed, you can try it at home, just get a bunch of bytes from urandom and feed them to various decoders, chances are low that you'll generate anything the UTF8 decoder will accept, chances are also low that you'll generate anything an ISO-8859 character set will reject.