News for hackers. Stuff that matters.
Re:APFS is not safe to use with names which have Unicode nor... by klodolph on Tuesday June 06, 2017 @10:02AM (#14497256)
NTFS, by comparison, does the "bag of wchar_t" instead of the "bag of byte" for filenames.
Re:APFS is not safe to use with names which have Unicode nor... by rbanffy on Tuesday June 06, 2017 @11:34AM (#14498000)
> NTFS, by comparison, does the "bag of wchar_t"

I threw up a little.

Re:APFS is not safe to use with names which have Unicode nor... by aidenn0 on Tuesday June 06, 2017 @12:18PM (#14498398)
This is largely due to microsoft being an early adopter of unicode. In the 90s, unicode was limited to 16 bits, so switching to fixed-length 16-bit characters made a lot of sense.

Of course, it didn't take long for that to change, but MS had already put a lot of resources into switching to UCS-2.

Re:APFS is not safe to use with names which have Unicode nor... by dietrichepp on Tuesday June 06, 2017 @12:45PM (#14498668)
This timeline also meant that wchar_t was fixed as 16 bits on Windows for compatibility, but ended up as 32 bits on macOS and Unix. It can't really be removed from the C or C++ standards, but there's nothing you can do with wchar_t that's both useful and portable.

Well-meaning developers learn how to support Unicode with wchar_t only to discover that wchar_t is the worst way to support Unicode.

https://www.moria.us/articles/wchar-is-a-historical-accident...

Re:APFS is not safe to use with names which have Unicode nor... by rspeer on Tuesday June 06, 2017 @02:41PM (#14499892)
It doesn't help that many developers are introduced to the basics of Unicode by a popular Joel on Software article from 2003, which recommended the use of wchar_t. [1]

It was a very important article for 2003, but it should be honorably retired.

Its plan of action isn't as good as the modern "UTF-8 Everywhere", and even its motivating examples are becoming less relevant: when was the last time you went to the 'Encoding' menu in your web browser and guessed which codepage the page author meant to use? When was the last time you worried about whether your e-mail was 7-bit clean?

[1] https://www.joelonsoftware.com/2003/10/08/the-absolute-minim...

Re:APFS is not safe to use with names which have Unicode nor... by cryptonector on Tuesday June 06, 2017 @01:24PM (#14499054)
HFS+'s use of NFD also comes from it being an early adopter of Unicode.

When I argued strenuously for n-p/n-i behavior in ZFS (see elsewhere in this thread) I was told that gee, Unicode doesn't describe n-i string comparison the way I was proposing, so we can't do it :( However, fast n-i does follow from the spec, so eventually it got done. I can't take credit for the code, though I did contribute an optimization.

Re:APFS is not safe to use with names which have Unicode nor... by cryptonector on Tuesday June 06, 2017 @12:44PM (#14498656)
"Bag of code units" would be a better description (if it's correct anyways).

This is not very good. You want normalization-preserving/insensitive behavior instead.

Re:APFS is not safe to use with names which have Unicode nor... by klodolph on Wednesday June 07, 2017 @12:59AM (#14503868)
What about "bag of wchar_t" doesn't preserve normalization? Or am I not sure what you're trying to say?

It's no more a "bag of code units" than Linux filesystems store a "bag of code units". Windows will barf back whatever wchar_t array you give it, just like Linux will barf back whatever char array you give it.

Re:APFS is not safe to use with names which have Unicode nor... by cryptonector on Wednesday June 07, 2017 @10:13AM (#14506587)
I'm saying "code unit" is the correct Unicode term for "8-bit bytes" (UTF-8) or "16-bit values" (UTF-16).
Re:APFS is not safe to use with names which have Unicode nor... by klodolph on Wednesday June 07, 2017 @09:00PM (#14511117)
You're right that "code unit" is the correct Unicode term for parts of encoded Unicode strings, which is exactly why it's the wrong term in this case, because the whole point is the filenames don't have to be Unicode.

So I can take the byte string [0xFF, 0xFF, 0xFF] and use that as a filename on Linux, or I can take the sequence of 16-bit values [0xD800, 0xD800, 0xD800] and use that as a filename on NTFS. They're not made of code units because they're not Unicode strings.