U+0031 1 [Nd] DIGIT ONE
U+0020 [Zs] SPACE
U+002E . [Po] FULL STOP
U+0020 [Zs] SPACE
U+6559 教 [Lo] CJK UNIFIED IDEOGRAPH-6559
U+6703 會 [Lo] CJK UNIFIED IDEOGRAPH-6703
U+7D44 組 [Lo] CJK UNIFIED IDEOGRAPH-7D44
U+7E54 織 [Lo] CJK UNIFIED IDEOGRAPH-7E54
U+003A : [Po] COLON
U+0020 [Zs] SPACE
U+F081 \uf081 [Co] <unknown>
U+5C0F 小 [Lo] CJK UNIFIED IDEOGRAPH-5C0F
U+6703 會 [Lo] CJK UNIFIED IDEOGRAPH-6703
U+003A : [Po] COLON
...
The anomalous character is U+F081, a character from the Private Use Area. TextWrangler is allowed to interpret it as whatever it wants, but I don't know why that would mess up all the following characters.Here's my theory. The text probably started out in the GBK encoding (used in mainland China). GBK has had different versions, which supported slightly different sets of characters. A number of these characters (decreasing as both GBK and Unicode updated) have no corresponding character in Unicode, and the standard thing to do when converting them to Unicode has been to convert them into Private Use characters.
So that probably happened to this one, which may have started as a rare and inconsistently-supported character.
Python's implementation of GBK (or GB18030) doesn't know what it is. So maybe what we need to do is flip through this Chinese technical standard [1], or maybe an older version of it, and track down which codepoint was historically mapped to U+F081 and what it is now and hahaha oh god
[1] https://archive.org/stream/GB18030-2005/GB%2018030-2005#page...
Try reading this in TextWrangler: 1 . 教會組織: 小會:代議長老郭倍宏
The box causes the follow characters to be unreadable - it gets interpreted as a half-character. Deleting it makes the text show correctly.
I tried it with ftfy, but it just copied the input through to the output.