The comparison with other filesystems does not hold since applications for other OSs have always been developed with no normalization at FS level, and hence it was done by the applications, or through the use of high-level OS APIs. Mac applications, on the other hand, expect it to be the responsibility of the filesystem. This is explained pretty clearly in the article.
On a side note, I wish Apple would have taken this opportunity to switch normalization from NFD to NFC, which basically everything else uses. The distinction causes complexity and often issues in software which share data between Apple platforms and other platforms, such as version control systems for instance.
EDIT: according to pilif's comment, they did, which is awesome!
> Mac applications, on the other hand, expect it to be the responsibility of the filesystem. This is explained pretty clearly in the article.
Again, the article made a huge sweeping claim without supporting it. That's simply not true in either way – many apps on every OS don't handle this at all, some handle it consistently everywhere, and what a “Mac application” means varies widely from “clean, modern Cocoa” to “uses a lot of C, etc. libraries”, “cross platform C++”, “C# port”, “Electron shell”, etc. You can't make any statement which is true for every single one of those categories, much less for every code path which eventually results in a filesystem call. I've run into cases where something mostly worked until you hit their integrated ZIP, Git/SVN, etc. support and found a new way that a filename was constructed.
My point wasn't that everything is fine but simply that this is complicated and no decision results in avoiding problems. Not normalizing allows for confusing visually-identical files; normalizing results in errors or data loss which will be blamed on the OS.
Looks to me like the article made claims and backed them up with examples and screenshots.
Rather than saying the article is wrong, can you demonstrate /why/ it is wrong? Using its examples and concerns (Finder, console, and scripts)?
I think you might to re-read my entire comment: note that I'm not arguing that the technical details are wrong, only that they're insufficient to support the huge “APFS is unusable” conclusion.
As previously noted, Windows and Linux work the same way and they are used by more people in individual non-English locales than the total number of Mac users. Would you say “NTFS is unusable by non-English users” is a useful statement?
There's plenty of room to say that a particular tool needs improvement, or that people making systems which copy or archive files should check for pathological cases, but it doesn't help anything to overstate the case so broadly.
It's not a problem on Windows or Linux filesystems because Windows and Linux don't provide a half-assed normalization scheme that lets me fairly easily create files that can't be accessed. If the Cocoa libraries did no normalization, then the resulting behavior might be obnoxious from a human-interface perspective, but I don't think the article would describe it as "little short of catastrophic".
I'm sitting here on my US English keyboard typing scancodes that look just like they did in 1990, so I'm not the best authority on how big of a problem it really is, but I'd guess it's going to result in a lot of bugs. Anyone who's ever tried to use a Mac with a case-sensitive HFS+ partition should be able to tell you that programmers can't even "normalize" their filenames consistently strictly within their native language.
This is only true if you're talking about the kernel APIs. Unfortunately, filenames come from a variety of sources and it's easy to find tools which inconsistently normalize them – e.g. simply copying and pasting a name from a Word doc, web page, etc. which has different normalization than whatever originally created the file – or which produce either duplicate error messages or confusing error messages because the normalization form used in a file doesn't match the normalization form written on disk.
I've encountered variations of this problem on all three systems. No approach is going to handle 100% of the filenames in the wild and all of them will require extra care in the user-interface which may or may not have been done – e.g. the Windows Explorer still provides no way to tell why Café.txt and Café.txt are not the same file – and fixing the cases where programs are internally inconsistent. APFS switching will expose some programs which were unsafe before but since it's consistent with the other common filesystems it'll remove the need for every archive, version control, etc. system to either special-case or break.
The reason it's not that expensive is: a) this requires no memory allocation, and b) most characters in most strings require no normalization!
Notionally you just look at pairs of next codepoints, and if the second one isn't combining and the first one is canonical for the chosen NF (a very fast check for ASCII!), then there's no need to normalize the first, otherwise you gather the combining codepoints and normalize, producing one normalized character and restarting the process where you left off. Most of the time the first codepoint requires no normalization, so the fast path is fast -- not as fast as a normal strcmp() or memcmp(), but still pretty fast.
Why? It's done once per open, not once per I/O.
In ZFS it's once per-open()/stat()/and so on. But still, not at all on readdir(), and anyways, it's highly optimized. For an all ASCII filename the slow path is never taken, and for a mostly ASCII filename the slow path is only taken for non-ASCII codepoints that are followed by combining codepoints (that check is itself a slower-than-the-fast path, but still faster than the slowest path).
Basically, it's noise.
Specifically, ZFS has a normalization-preserving, normalization-insensitive behavior -- a lot like case-preserving but case-insensitive behavior, but for normalization forms rather than case.
The way this works is that there's a) a string comparison function that can provide normalization- and/or case-insensitive comparisons, b) a character-at-a-time normalization function of sorts used for directory hashing (since ZFS hashes directories).
This is much better than normalizing on create, which is destructive and obnoxious when the form you normalize to is not the more common form and you live in a sea of applications that don't do normalization.
That's not a question of being true or not but referring to different things. The distinction I was trying to make is that HFS+ will force every filename into NFD. With ZFS, the filename as received from the APIs should be the same byte sequence which was used to create it and that avoids an entire category of bugs where e.g. a program writes a file and fails to find it in a later readdir() call. As an example, Subversion and Git both had numerous bug reports over the years where it was impossible to simply checkout a repo containing a file which used a different normalization form.
Say you create a file with a name that has different NFC and NFD forms, and you create it with the NFC form. Then you go try to create it with the NFD form, well, if doing an exclusive create (O_EXCL) then you'll get EEXIST, else you'll open the existing file.
I threw up a little.
Of course, it didn't take long for that to change, but MS had already put a lot of resources into switching to UCS-2.
Well-meaning developers learn how to support Unicode with wchar_t only to discover that wchar_t is the worst way to support Unicode.
https://www.moria.us/articles/wchar-is-a-historical-accident...
It was a very important article for 2003, but it should be honorably retired.
Its plan of action isn't as good as the modern "UTF-8 Everywhere", and even its motivating examples are becoming less relevant: when was the last time you went to the 'Encoding' menu in your web browser and guessed which codepage the page author meant to use? When was the last time you worried about whether your e-mail was 7-bit clean?
[1] https://www.joelonsoftware.com/2003/10/08/the-absolute-minim...
When I argued strenuously for n-p/n-i behavior in ZFS (see elsewhere in this thread) I was told that gee, Unicode doesn't describe n-i string comparison the way I was proposing, so we can't do it :( However, fast n-i does follow from the spec, so eventually it got done. I can't take credit for the code, though I did contribute an optimization.
This is not very good. You want normalization-preserving/insensitive behavior instead.
It's no more a "bag of code units" than Linux filesystems store a "bag of code units". Windows will barf back whatever wchar_t array you give it, just like Linux will barf back whatever char array you give it.
So I can take the byte string [0xFF, 0xFF, 0xFF] and use that as a filename on Linux, or I can take the sequence of 16-bit values [0xD800, 0xD800, 0xD800] and use that as a filename on NTFS. They're not made of code units because they're not Unicode strings.
Sequence of bytes. "spiroagnew.txt" and "xtt.growapenis" are not the same filename.
You need to continue to write this on APFS. Nothing really changed in that regard. It's now just also possible to store denormalized filenames.
My position on this is mixed. I've had to write code to deal with normalization changes in archives and it's quite tedious. The HFS+ approach of normalization is in many ways the best choice considered in isolation but in practice it's really expensive to support since most filesystems, tools, APIs, etc. predate Unicode and everyone else chose the bag of bytes approach.