Re:Data Processing: PLINQ, Parallelism and Performance
by ori_b on Friday January 07, 2011 @01:09PM
(#2080405)
I think you misread - "assures", not "assumes". UTF8 guarantees that any valid ASCII char is NOT part of a UTF8 encoded multibyte char.
Re:Data Processing: PLINQ, Parallelism and Performance
by ot on Friday January 07, 2011 @02:46PM
(#2080859)
I think you are right.
I thought that the author was meaning that it is possible to use \n and \t in values because UTF8 would encode them in multibyte sequences (like Modified UTF8 encodes \0 as 0xC0,0x80).
What he was actually meaning is that if a codepoint is > 127 its multibyte encoding won't contain any \n or \t.
Sorry for the confusion.
> Each place name is represented by a UTF-8 (en.wikipedia.org/wiki/UTF-8) text line record (variable length) with more than 15 tab-separated data columns. Note: The UTF-8 encoding assures that a tab (0x9) or line feed (0xA) value won’t occur as part of a multi-byte sequence; this is essential for several implementations.
What? I guess that they use a longer (2-byte?) encoding for those codepoints, but from the very same wikipedia page that they link:
> a sequence that decodes to a value that should use a shorter sequence (an "overlong form") [is invalid]
...
> Implementations of the decoding algorithm MUST protect against decoding invalid sequences
Are they advising to use an invalid and potentially broken UTF8 encoding?