News for hackers. Stuff that matters.
Re:What programmers need to know about encodings and charset... by BiteCode_dev on Saturday August 15, 2020 @02:48AM (#24166613)
In a sense, it is simple: simple to understand.

Not simple to solve.

Like to win a race Usain Bolt, it's simple, run faster!

Fortunately, Python is well equipped for that. If you open a file with Python that you know it might contains mixed encoded text, you can use try/except to inform the user or open in binary mode, and just store the binary.

But my favorite way of doing it is:

    open('file', error=strategy)
Strategy can be:

- "ignore": undecodable text is skipped

- "replace": undecodable text is replace with "?"

- "surrogateescape" (you need to use utf8): undecodable text is decoded to a special representation which makes no human sense, but can en rencoded back to it's original value.

It's kinda ironic because people bashed Python for separating bytes/text, forcing them to deal with encoding correctly in Python 3. After all, this problem of "slamming bytes together" comes from languages that treat text as a bytes array, allowing this stupid mistake.

Re:What programmers need to know about encodings and charset... by at_a_remove on Saturday August 15, 2020 @03:12AM (#24166718)
Yeah, that was more or less my approach to analysis for these files. Essentially it was an export that was quasi-aware of Unicode and dumped out certain fields (of course they had to be variable length) in their original encodings, whatever they were. I got more than a few of these.

Unicode is great, as long as everyone upstream follows all of the rules and nothing goes wrong.