Re:cuneicode, and the Future of Text in C
by kevin_thibedeau on Wednesday June 07, 2023 @12:12PM
(#36228823)
UTF-8 should never contain surrogates as they serve no purpose when all code points can be encoded directly. That doesn't prevent programmers doing stupid things with them but that's not a Unicode problem.
[ Parent
| on HN ]
Correct – but that’s exactly why you can’t just say “all strings are UTF-8, we’re done here”. Because Windows filenames can contain unpaired surrogates, which don’t have a sensible UTF-8 translation. (Unix has a similar problem with filenames being arbitrary bytes that may not be valid UTF-8.)
Note that Rust has OsString to handle both of these cases, but it’s somewhat hard to use.