News for hackers. Stuff that matters.
Re:APFS does not normalize Unicode filenames by ridiculous_fish on Saturday March 25, 2017 @01:16AM (#13954222)
This feels in some sense like punting the problem. What exactly should the presentation layer do when presenting two files, where the first is named with a precomposed character sequence, and the other has the same name but decomposed? Surface the normalization form the user? Uh, no...

The more fundamental question is whether filenames are under control of the user or the system. The answer today is "both": there's blessed paths /System/Library... and non-blessed paths like ~/Documents/Pokemon.txt. Addressing this properly means reifying that distinction: making apps always be explicit about whether they're working with the user's or the filesystem's view of a file.

While we're here, why the should the user even have to name every file? Do you name every piece of junk mail on your desk/kitchen island?

The ideal for the user is something like tagging, where naming stuff is optional, names are forgiving and perhaps not unique, and files are not restricted to being in one directory. Meanwhile, filesystem names are an implementation detail and the filesystem enjoys Unicode ignorance. Spotlight moved a bit in this direction, but the end goal is still awfully far away.

Re:APFS does not normalize Unicode filenames by comex on Saturday March 25, 2017 @03:34AM (#13954504)
> What exactly should the presentation layer do when presenting two files, where the first is named with a precomposed character sequence, and the other has the same name but decomposed?

Show two files with apparently identical names. Is that so surprising? There are many many ways for two different Unicode strings to look visually identical (or near-identical) even if they aren't equivalent under normalization. For that matter, even if you were limited to plain keyboard ASCII, you could have two really long filenames that only differ in one character, which are effectively indistinguishable without massive hair-pulling. If you want to be robust, there's no way around having means to identify files other than names.

(I agree with your sentiment about tagging.)

Re:APFS does not normalize Unicode filenames by askvictor on Saturday March 25, 2017 @05:29AM (#13954756)
Google drive allows you to have identically named files in the same directory. Doesn't quite map onto a desktop filesystem, but the filename is not a unique identifier, but just a convenience for the user or application.
Re:APFS does not normalize Unicode filenames by jameshart on Saturday March 25, 2017 @08:22AM (#13955271)
Even with normalization, you can just sprinkle an arbitrary selection of non breaking zero width space characters into a string and create an infinite variety of identical-looking but distinct codepoint sequences. Treating filenames as bundles of bytes is fine, but at that point it seems dangerous to assume that interpreting them as encoded strings is a valid way of distinguishing them.
Re:APFS does not normalize Unicode filenames by rurban on Saturday March 25, 2017 @04:33AM (#13954628)
You are talking about confusables. Unfortunately normalization cannot handle this. Only certain apps treat confusables as confusables: email, web browsers, dns services, usually via libidn.
Re:APFS does not normalize Unicode filenames by kbenson on Saturday March 25, 2017 @04:34PM (#13957442)
> For that matter, even if you were limited to plain keyboard ASCII, you could have two really long filenames that only differ in one character, which are effectively indistinguishable without massive hair-pulling.

My initial response was to not present confusing data to the user, but then I thought of exactly this. In many fonts, lowercase "L" (l) and uppercase "i" (I) look the same. I'm sure in some fonts they actually are identical, rather than just being nearly indistinguishable to the human eye.

To the discussion is not about whether we should present confusing identifiers to users, but at what point if it becomes more common does it cause enough of a problem to be a bad choice, and what techniques are there to mitigate it?

Re:APFS does not normalize Unicode filenames by panic on Saturday March 25, 2017 @02:12AM (#13954336)
The more fundamental question is whether filenames are under control of the user or the system. The answer today is "both": there's blessed paths /System/Library... and non-blessed paths like ~/Documents/Pokemon.txt. Addressing this properly means reifying that distinction: making apps always be explicit about whether they're working with the user's or the filesystem's view of a file.

This distinction is important for other reasons, too. Should this file be indexed for search? Should it store an access time to support features like "recently opened"?

On the other side, we want to know which blessed paths "belong" to which system application or service so we can sync them, protect access to them, and uninstall them with their owners.

Maybe the user-visible document store should be a separate mechanism entirely from low-level app/service data persistence.

Re:APFS does not normalize Unicode filenames by nradov on Saturday March 25, 2017 @10:17AM (#13955736)
Microsoft tried to do something like your ideal with WinFS but couldn't make it work and abandoned the project.
Re:APFS does not normalize Unicode filenames by Ericson2314 on Saturday March 25, 2017 @01:30AM (#13954258)
I'd argue: Use a langauge with generics and make the path type a ((de-)serializable) black box.

You still need to make a decision in the end, but are much more free to change it.

Re:APFS does not normalize Unicode filenames by jstimpfle on Saturday March 25, 2017 @06:40AM (#13954943)
This is not a language issue.
Re:APFS does not normalize Unicode filenames by Ericson2314 on Saturday March 25, 2017 @10:38AM (#13955858)
I didn't say it was. I said generics would help one easily switch their choice on the different semantics, not avoid choosing at all.
Re:APFS does not normalize Unicode filenames by jstimpfle on Saturday March 25, 2017 @10:39AM (#13955865)
Many, or most, languages already do that by representing filenames as native strings (which are usually abstract "Unicode" strings).
Re:APFS does not normalize Unicode filenames by Ericson2314 on Saturday March 25, 2017 @11:50PM (#13959037)
I want to swap the path type used by the FS implementation. This is totally distinct from a langauge providing some string type which while abstract is nevertheless fixed and unchangable.
Re:APFS does not normalize Unicode filenames by daenney on Saturday March 25, 2017 @07:35AM (#13955113)
Both Objective-C and Swift have (a form of) generics on board.