Why to normalize Unicode strings

Re:Why to normalize Unicode strings by ucarion on Wednesday March 13, 2019 @04:25PM (#19383371)

This comes up because many implementations of "UTF-16" are in fact naive, and just operate within the mentality of UCS-2, where everything is just a 16-bit number.

This is relevant within this discussion because we were talking about filesystems, and Windows stores file names as an arbitrary array of 16-bit numbers, but UTF-16 has additional requirements that Windows file names do not satisfy.

When I say UCS-2, I just mean pretending that UTF-16 is short for `[]uint16`, and not worrying about surrogate pairs.

[ Parent | on HN ]

Re:Why to normalize Unicode strings by jrochkind1 on Wednesday March 13, 2019 @04:57PM (#19383679)

I think we can all agree that you should never be applying any normalization to bytes representing a file system path, unless you are doing it according to specifications for that particular file system+OS combo.

file system paths are tricky with non-ascii, that is for sure! Especially if you are trying to be agnostic to OS/filesystem. They are an edge case.

[ Parent | on HN ]