News for hackers. Stuff that matters.

A comment on Why to normalize Unicode strings. Parent comment

Re:Why to normalize Unicode strings by happytoexplain on Wednesday March 13, 2019 @02:17PM (#19381702)
UCS-2 refers to the UTF-16 encoding of the first plane, defined when there was only one plane's worth of assignable space. UTF-16 extends UCS-2, and defines the reserved surrogates, thereby introducing the possibility of invalid surrogate usage. The parent post is referring to that (so it would be more accurate for the parent post to have said "UTF-16").
[ Parent | on HN ]
Re:Why to normalize Unicode strings by jrochkind1 on Wednesday March 13, 2019 @03:38PM (#19382785)
OK, thanks.

You can certainly have invalid bytes for UTF-8 too. I think in any unicode encoding, not every possible byte sequence is a valid representation of unicode codepoints. It doesn't require UCS-2/UTF-16.

Here's an invalid byte sequence in UTF-8: "\xc3\x28"

[ Parent | on HN ]
Re:Why to normalize Unicode strings by happytoexplain on Wednesday March 13, 2019 @03:43PM (#19382858)
True, though the GP's example was somewhat unique in that UTF-16 created the possibility of invalid combinations at the code point level, rather than only the byte level.
[ Parent | on HN ]