News for hackers. Stuff that matters.
Re:APFS is not safe to use with names which have Unicode nor... by acdha on Tuesday June 06, 2017 @07:18AM (#14496370)
There's a potentially useful discussion to be had on normalization but the title is pure clickbait hyperbole. HFS+ is the only filesystem in common use which performs Unicode normalization and a statement that bold would require at least some evidence that Windows, Linux, etc. are only usable by English speakers.

My position on this is mixed. I've had to write code to deal with normalization changes in archives and it's quite tedious. The HFS+ approach of normalization is in many ways the best choice considered in isolation but in practice it's really expensive to support since most filesystems, tools, APIs, etc. predate Unicode and everyone else chose the bag of bytes approach.

Re:APFS is not safe to use with names which have Unicode nor... by catwell on Tuesday June 06, 2017 @08:18AM (#14496624)
Normalization is very expensive and does not belong at the FS level. This kills performance for some classes of applications.

The comparison with other filesystems does not hold since applications for other OSs have always been developed with no normalization at FS level, and hence it was done by the applications, or through the use of high-level OS APIs. Mac applications, on the other hand, expect it to be the responsibility of the filesystem. This is explained pretty clearly in the article.

On a side note, I wish Apple would have taken this opportunity to switch normalization from NFD to NFC, which basically everything else uses. The distinction causes complexity and often issues in software which share data between Apple platforms and other platforms, such as version control systems for instance.

EDIT: according to pilif's comment, they did, which is awesome!

Re:APFS is not safe to use with names which have Unicode nor... by acdha on Tuesday June 06, 2017 @10:04AM (#14497264)
Do you have any recent benchmarks showing a significant impact from normalization? I haven't seen that on anything in at least a decade and that was simply Red Hat shipping an ancient and completely unoptimized libicu.

> Mac applications, on the other hand, expect it to be the responsibility of the filesystem. This is explained pretty clearly in the article.

Again, the article made a huge sweeping claim without supporting it. That's simply not true in either way – many apps on every OS don't handle this at all, some handle it consistently everywhere, and what a “Mac application” means varies widely from “clean, modern Cocoa” to “uses a lot of C, etc. libraries”, “cross platform C++”, “C# port”, “Electron shell”, etc. You can't make any statement which is true for every single one of those categories, much less for every code path which eventually results in a filesystem call. I've run into cases where something mostly worked until you hit their integrated ZIP, Git/SVN, etc. support and found a new way that a filename was constructed.

My point wasn't that everything is fine but simply that this is complicated and no decision results in avoiding problems. Not normalizing allows for confusing visually-identical files; normalizing results in errors or data loss which will be blamed on the OS.

Re:APFS is not safe to use with names which have Unicode nor... by vintagedave on Tuesday June 06, 2017 @10:36AM (#14497507)
> Again, the article made a huge sweeping claim without supporting it.

Looks to me like the article made claims and backed them up with examples and screenshots.

Rather than saying the article is wrong, can you demonstrate /why/ it is wrong? Using its examples and concerns (Finder, console, and scripts)?

Re:APFS is not safe to use with names which have Unicode nor... by acdha on Tuesday June 06, 2017 @12:19PM (#14498411)
> Rather than saying the article is wrong, can you demonstrate /why/ it is wrong?

I think you might to re-read my entire comment: note that I'm not arguing that the technical details are wrong, only that they're insufficient to support the huge “APFS is unusable” conclusion.

As previously noted, Windows and Linux work the same way and they are used by more people in individual non-English locales than the total number of Mac users. Would you say “NTFS is unusable by non-English users” is a useful statement?

There's plenty of room to say that a particular tool needs improvement, or that people making systems which copy or archive files should check for pathological cases, but it doesn't help anything to overstate the case so broadly.

Re:APFS is not safe to use with names which have Unicode nor... by deong on Tuesday June 06, 2017 @01:37PM (#14499203)
The issue isn't a "bag-of-bytes" filename model. The issue is a "bag-of-bytes" filename model combined with an inconsistent normalization scheme.

It's not a problem on Windows or Linux filesystems because Windows and Linux don't provide a half-assed normalization scheme that lets me fairly easily create files that can't be accessed. If the Cocoa libraries did no normalization, then the resulting behavior might be obnoxious from a human-interface perspective, but I don't think the article would describe it as "little short of catastrophic".

I'm sitting here on my US English keyboard typing scancodes that look just like they did in 1990, so I'm not the best authority on how big of a problem it really is, but I'd guess it's going to result in a lot of bugs. Anyone who's ever tried to use a Mac with a case-sensitive HFS+ partition should be able to tell you that programmers can't even "normalize" their filenames consistently strictly within their native language.

Re:APFS is not safe to use with names which have Unicode nor... by acdha on Tuesday June 06, 2017 @02:30PM (#14499776)
> It's not a problem on Windows or Linux filesystems because Windows and Linux don't provide a half-assed normalization scheme that lets me fairly easily create files that can't be accessed

This is only true if you're talking about the kernel APIs. Unfortunately, filenames come from a variety of sources and it's easy to find tools which inconsistently normalize them – e.g. simply copying and pasting a name from a Word doc, web page, etc. which has different normalization than whatever originally created the file – or which produce either duplicate error messages or confusing error messages because the normalization form used in a file doesn't match the normalization form written on disk.

I've encountered variations of this problem on all three systems. No approach is going to handle 100% of the filenames in the wild and all of them will require extra care in the user-interface which may or may not have been done – e.g. the Windows Explorer still provides no way to tell why Café.txt and Café.txt are not the same file – and fixing the cases where programs are internally inconsistent. APFS switching will expose some programs which were unsafe before but since it's consistent with the other common filesystems it'll remove the need for every archive, version control, etc. system to either special-case or break.

Re:APFS is not safe to use with names which have Unicode nor... by cryptonector on Tuesday June 06, 2017 @12:32PM (#14498554)
Normalization is not that expensive if you only care to do normalization-insensitive string comparison and string hashing.

The reason it's not that expensive is: a) this requires no memory allocation, and b) most characters in most strings require no normalization!

Notionally you just look at pairs of next codepoints, and if the second one isn't combining and the first one is canonical for the chosen NF (a very fast check for ASCII!), then there's no need to normalize the first, otherwise you gather the combining codepoints and normalize, producing one normalized character and restarting the process where you left off. Most of the time the first codepoint requires no normalization, so the fast path is fast -- not as fast as a normal strcmp() or memcmp(), but still pretty fast.

Re:APFS is not safe to use with names which have Unicode nor... by Animats on Tuesday June 06, 2017 @02:59PM (#14500083)
Normalization is very expensive and does not belong at the FS level. This kills performance for some classes of applications.

Why? It's done once per open, not once per I/O.

Re:APFS is not safe to use with names which have Unicode nor... by cryptonector on Tuesday June 06, 2017 @05:36PM (#14501588)
In HFS+ it's even only done once per-create, so pretty cheap.

In ZFS it's once per-open()/stat()/and so on. But still, not at all on readdir(), and anyways, it's highly optimized. For an all ASCII filename the slow path is never taken, and for a mostly ASCII filename the slow path is only taken for non-ASCII codepoints that are followed by combining codepoints (that check is itself a slower-than-the-fast path, but still faster than the slowest path).

Basically, it's noise.

Re:APFS is not safe to use with names which have Unicode nor... by cryptonector on Tuesday June 06, 2017 @12:28PM (#14498512)
Not true! ZFS also normalizes. However, while HFS+ normalizes on _create_ (bad), ZFS normalizes on lookup (good).

Specifically, ZFS has a normalization-preserving, normalization-insensitive behavior -- a lot like case-preserving but case-insensitive behavior, but for normalization forms rather than case.

The way this works is that there's a) a string comparison function that can provide normalization- and/or case-insensitive comparisons, b) a character-at-a-time normalization function of sorts used for directory hashing (since ZFS hashes directories).

This is much better than normalizing on create, which is destructive and obnoxious when the form you normalize to is not the more common form and you live in a sea of applications that don't do normalization.

Re:APFS is not safe to use with names which have Unicode nor... by acdha on Tuesday June 06, 2017 @02:47PM (#14499949)
> Not true! ZFS also normalizes. However, while HFS+ normalizes on _create_ (bad), ZFS normalizes on lookup (good).

That's not a question of being true or not but referring to different things. The distinction I was trying to make is that HFS+ will force every filename into NFD. With ZFS, the filename as received from the APIs should be the same byte sequence which was used to create it and that avoids an entire category of bugs where e.g. a program writes a file and fails to find it in a later readdir() call. As an example, Subversion and Git both had numerous bug reports over the years where it was impossible to simply checkout a repo containing a file which used a different normalization form.

Re:APFS is not safe to use with names which have Unicode nor... by cryptonector on Tuesday June 06, 2017 @04:06PM (#14500824)
Yes, exactly. You can't have those buggy behaviors in git/svn/hg/whatever if the FS is form-preserving.
Re:APFS is not safe to use with names which have Unicode nor... by ridiculous_fish on Tuesday June 06, 2017 @12:39PM (#14498608)
That's fascinating. What happens on lookup if there's two different files with canonically equivalent names?
Re:APFS is not safe to use with names which have Unicode nor... by cryptonector on Tuesday June 06, 2017 @12:49PM (#14498706)
Great question! You can't have two such files!

Say you create a file with a name that has different NFC and NFD forms, and you create it with the NFC form. Then you go try to create it with the NFD form, well, if doing an exclusive create (O_EXCL) then you'll get EEXIST, else you'll open the existing file.

Re:APFS is not safe to use with names which have Unicode nor... by throwaway2048 on Tuesday June 06, 2017 @03:42PM (#14500537)
so it infact does do normalization on file create then? How is it managing this without doing so?
Re:APFS is not safe to use with names which have Unicode nor... by cryptonector on Tuesday June 06, 2017 @04:02PM (#14500778)
It normalizes on LOOKUP, not create. On create it preserves the original form. Think of this as like case-preserving/case-insensitive behavior, but for form rather than case.
Re:APFS is not safe to use with names which have Unicode nor... by klodolph on Tuesday June 06, 2017 @10:02AM (#14497256)
NTFS, by comparison, does the "bag of wchar_t" instead of the "bag of byte" for filenames.
Re:APFS is not safe to use with names which have Unicode nor... by rbanffy on Tuesday June 06, 2017 @11:34AM (#14498000)
> NTFS, by comparison, does the "bag of wchar_t"

I threw up a little.

Re:APFS is not safe to use with names which have Unicode nor... by aidenn0 on Tuesday June 06, 2017 @12:18PM (#14498398)
This is largely due to microsoft being an early adopter of unicode. In the 90s, unicode was limited to 16 bits, so switching to fixed-length 16-bit characters made a lot of sense.

Of course, it didn't take long for that to change, but MS had already put a lot of resources into switching to UCS-2.

Re:APFS is not safe to use with names which have Unicode nor... by dietrichepp on Tuesday June 06, 2017 @12:45PM (#14498668)
This timeline also meant that wchar_t was fixed as 16 bits on Windows for compatibility, but ended up as 32 bits on macOS and Unix. It can't really be removed from the C or C++ standards, but there's nothing you can do with wchar_t that's both useful and portable.

Well-meaning developers learn how to support Unicode with wchar_t only to discover that wchar_t is the worst way to support Unicode.

https://www.moria.us/articles/wchar-is-a-historical-accident...

Re:APFS is not safe to use with names which have Unicode nor... by rspeer on Tuesday June 06, 2017 @02:41PM (#14499892)
It doesn't help that many developers are introduced to the basics of Unicode by a popular Joel on Software article from 2003, which recommended the use of wchar_t. [1]

It was a very important article for 2003, but it should be honorably retired.

Its plan of action isn't as good as the modern "UTF-8 Everywhere", and even its motivating examples are becoming less relevant: when was the last time you went to the 'Encoding' menu in your web browser and guessed which codepage the page author meant to use? When was the last time you worried about whether your e-mail was 7-bit clean?

[1] https://www.joelonsoftware.com/2003/10/08/the-absolute-minim...

Re:APFS is not safe to use with names which have Unicode nor... by cryptonector on Tuesday June 06, 2017 @01:24PM (#14499054)
HFS+'s use of NFD also comes from it being an early adopter of Unicode.

When I argued strenuously for n-p/n-i behavior in ZFS (see elsewhere in this thread) I was told that gee, Unicode doesn't describe n-i string comparison the way I was proposing, so we can't do it :( However, fast n-i does follow from the spec, so eventually it got done. I can't take credit for the code, though I did contribute an optimization.

Re:APFS is not safe to use with names which have Unicode nor... by cryptonector on Tuesday June 06, 2017 @12:44PM (#14498656)
"Bag of code units" would be a better description (if it's correct anyways).

This is not very good. You want normalization-preserving/insensitive behavior instead.

Re:APFS is not safe to use with names which have Unicode nor... by klodolph on Wednesday June 07, 2017 @12:59AM (#14503868)
What about "bag of wchar_t" doesn't preserve normalization? Or am I not sure what you're trying to say?

It's no more a "bag of code units" than Linux filesystems store a "bag of code units". Windows will barf back whatever wchar_t array you give it, just like Linux will barf back whatever char array you give it.

Re:APFS is not safe to use with names which have Unicode nor... by cryptonector on Wednesday June 07, 2017 @10:13AM (#14506587)
I'm saying "code unit" is the correct Unicode term for "8-bit bytes" (UTF-8) or "16-bit values" (UTF-16).
Re:APFS is not safe to use with names which have Unicode nor... by klodolph on Wednesday June 07, 2017 @09:00PM (#14511117)
You're right that "code unit" is the correct Unicode term for parts of encoded Unicode strings, which is exactly why it's the wrong term in this case, because the whole point is the filenames don't have to be Unicode.

So I can take the byte string [0xFF, 0xFF, 0xFF] and use that as a filename on Linux, or I can take the sequence of 16-bit values [0xD800, 0xD800, 0xD800] and use that as a filename on NTFS. They're not made of code units because they're not Unicode strings.

Re:APFS is not safe to use with names which have Unicode nor... by wnoise on Tuesday June 06, 2017 @08:29PM (#14502754)
> everyone else chose the bag of bytes approach.

Sequence of bytes. "spiroagnew.txt" and "xtt.growapenis" are not the same filename.

Re:APFS is not safe to use with names which have Unicode nor... by the_mitsuhiko on Tuesday June 06, 2017 @11:52AM (#14498146)
> I've had to write code to deal with normalization changes in archives and it's quite tedious.

You need to continue to write this on APFS. Nothing really changed in that regard. It's now just also possible to store denormalized filenames.

Re:APFS is not safe to use with names which have Unicode nor... by acdha on Tuesday June 06, 2017 @12:21PM (#14498429)
Exactly - and that matters to me in cases where the ability to say that someone copying files won't change the checksum on a manifest, which users find extremely confusing.