News for hackers. Stuff that matters.
Re:APFS is not safe to use with names which have Unicode nor... by cstross on Tuesday June 06, 2017 @07:17AM (#14496363)
The real problem isn't whether filename normalization is a good/bad thing, the problem is Apple used to do it one way and is now switching without warning to doing it the other way.

It's the logical end product of the odyssey from Apple's original philosophy of a resource and data fork model for files to the UNIX stream-of-bytes model for files. The UNIX model traditionally kept metadata about files separate (anyone else remember naughtily hand-editing directory files with sed(1), back in the day?) but gradually picked up a bunch of new stuff that has to be stored somewhere. Meanwhile, the Mac platform adopted UNIX binaries with the move to NeXTStep underpinnings (we're going back to the late 90s here, and the adoption of OSX over traditional MacOS), obviating the need for the original resource bundle, which got rid of those annoying errors about missing bundle bits but left us with a legacy of .DS_Store turdlets in every directory to hold file metadata that was formerly stored in the resource fork.

As a destination it's laudable — design consistency is almost always laudable — but it's the end goal of a very messy process and it looks like APFS isn't quite there yet; NSFileManager and NSURL need some way of distinguishing files with different unicode representations, and the Finder in particular needs to be robust. I'm guessing this is going to be fixed when High Sierra finally ships, but isn't supported in Sierra at present, hence the OP's alarm at the situation.

Re:APFS is not safe to use with names which have Unicode nor... by mchanson on Tuesday June 06, 2017 @08:22AM (#14496638)
"without warning"? Not really. This has been discussed to death on ATP last year.

Also, in regards to the headline, there are tens or maybe over a hundred million non-english speakers using iOS already running APFS...

Re:APFS is not safe to use with names which have Unicode nor... by TorKlingberg on Tuesday June 06, 2017 @08:38AM (#14496713)
On iOS the user doesn't get much direct access to the file system.
Re:APFS is not safe to use with names which have Unicode nor... by the_mitsuhiko on Tuesday June 06, 2017 @08:44AM (#14496751)
iOS never exposes low-level filenames to the user.
Re:APFS is not safe to use with names which have Unicode nor... by JoachimS on Tuesday June 06, 2017 @09:09AM (#14496897)
One of the new features in iOS 11 presented yesterday is the File App. It provides access to all files (at some level) in the system and on networked systems including iCloud, Dropbox etc.
Re:APFS is not safe to use with names which have Unicode nor... by the_mitsuhiko on Tuesday June 06, 2017 @09:17AM (#14496941)
That only ever surfaces filenames through cocoa APIs that underwent normalization.
Re:APFS is not safe to use with names which have Unicode nor... by cryptonector on Tuesday June 06, 2017 @12:43PM (#14498645)
Did anyone mention normalization-preserving/normalization-insensitive behavior? That's what ZFS does, and it's great!
Re:APFS is not safe to use with names which have Unicode nor... by kstrauser on Tuesday June 06, 2017 @01:34PM (#14499171)
> Did anyone mention normalization-preserving/normalization-insensitive behavior?

You did, 11 times so far this thread. I counted.

[dead]
[dead]
Re:APFS is not safe to use with names which have Unicode nor... by d0ugal on Tuesday June 06, 2017 @03:06PM (#14500169)
Is this reddit?
Re:APFS is not safe to use with names which have Unicode nor... by foota on Tuesday June 06, 2017 @01:10PM (#14498914)
Is that not just treating the filename as a byte sequence?
Re:APFS is not safe to use with names which have Unicode nor... by cryptonector on Tuesday June 06, 2017 @01:15PM (#14498962)
Almost! But not quite: it aliases some possible filenames, though only they are valid UTF-8 encodings (or UTF-16 say, if one did this on Windows). In particular, it prevents the existence of two files with equivalent names.
Re:APFS is not safe to use with names which have Unicode nor... by derefr on Tuesday June 06, 2017 @02:29PM (#14499765)
In other words, like most RDBMSes, ZFS differentiates "field type" from "collation."

"Field type" in an RDBMS controls what can be written (e.g. "valid UTF8 strings"), what will be read back (e.g. the use of the Unicode replacement character), and what special values like NULL will cast to.

An RDBMS field's collation controls how values in the field will compare for equality, and what will happen when you sort on that field.

Unique constraints in RDBMSes function on top of collation—so if the collation says two values are equivalent, the unique constraint will prevent you from inserting the new one.

Great properties, if you can get them. It's too bad that the filesystem API is as low-level as it is, actually; if POSIX had some concept of "readdir(2) pre-sorted by a given field" like e.g. Windows does, then you could require that userland programs rely on FS-level collation rather than allowing them to collate the resulting values however they feel like (usually meaning "naively.")

Re:APFS is not safe to use with names which have Unicode nor... by cryptonector on Tuesday June 06, 2017 @03:59PM (#14500741)
Well, readdir() in ZFS does not produce outputs in any order, since directories are just plain hash tables.

And the n-i string comparison function produces a boolean if I remember correctly, not a trinary. So it doesn't define a collation. But it could define a collation, that's true.

The problem with moving sorting into the kernel is that you now need to have the collations there (English? French? something else? "Unicode" is not enough), and user-land needs to tell the kernel what collation to use for any given process or thread or system call. That's ETOOMUCHWORK for everyone, so it doesn't happen.

Re:APFS is not safe to use with names which have Unicode nor... by adrianratnapala on Wednesday June 07, 2017 @06:16AM (#14504984)
Indeed!

Moreover, like it should be EBADIDEA.

Re:APFS is not safe to use with names which have Unicode nor... by foota on Tuesday June 06, 2017 @01:23PM (#14499039)
Interesting, so it recognizes different encodings of the same Unicode code points?
Re:APFS is not safe to use with names which have Unicode nor... by cryptonector on Tuesday June 06, 2017 @01:38PM (#14499214)
Yes. Well, it recognizes different forms of the same characters. The encoding is and must be UTF-8.
Re:APFS is not safe to use with names which have Unicode nor... by foota on Tuesday June 06, 2017 @01:24PM (#14499052)
Or does it just map code points that are identical to each other?
Re:APFS is not safe to use with names which have Unicode nor... by cryptonector on Tuesday June 06, 2017 @01:39PM (#14499226)
Yes. But only for the purpose of string comparison and hashing (ZFS hashes directories for fast lookups).
Re:APFS is not safe to use with names which have Unicode nor... by kstrauser on Wednesday June 07, 2017 @01:16PM (#14508085)
(BTW, I found your comments interesting, even as I laughed at how often you were making them. Thanks for contributing, and sorry for getting us both flagged (for IMHO the goofiest possible reason).)
Re:APFS is not safe to use with names which have Unicode nor... by vetinari on Tuesday June 06, 2017 @07:32AM (#14496436)
Technically, .DS_Store is just Finder data - where is the windows displaying the directory, it's dimensions, listing mode, etc.

The former resource fork is in ._$filename. You won't see it, unless you copy the file to smb share or zip it.

Re:APFS is not safe to use with names which have Unicode nor... by klodolph on Tuesday June 06, 2017 @10:00AM (#14497245)
The former resource fork not in ._$filename unless the underlying filesystem does not support it. On HFS+ volumes, which is what 99%+ of Mac users use, there will be no ._$filename file.
Re:APFS is not safe to use with names which have Unicode nor... by vetinari on Tuesday June 06, 2017 @10:20AM (#14497376)
You are correct regarding the resource fork support in the fs and it's visibility, maybe I didn't express myself as exactly as I should.

However, 99%+ of Mac users do use FAT-formatted USB sticks, ZIP files, or other fs/mechanism/whatever that does not support resource forks where the compatibility littering kicks in, so they, or the people they share their files with, will see ._$filename files too.

On the other hand, Finder litters with .DS_Store files everywhere, even on HFS+.

Re:APFS is not safe to use with names which have Unicode nor... by klodolph on Tuesday June 06, 2017 @10:36AM (#14497511)
Most files don't have resource forks though, so the ._$filename doesn't store resource fork data in most cases. It usually just stores metadata.

I really doubt that 99%+ of Mac users use flash drives / zip files, though. I suspect that well more than 1% of users never use anything like that. A lot of people only have a single computer and share through e.g. just mailing files to people or using Dropbox, or not even that.

Re:APFS is not safe to use with names which have Unicode nor... by vetinari on Tuesday June 06, 2017 @11:15AM (#14497859)
While resource forks are deprecated and on the retreat, extended attributes are a new hotness, used extensively (just download a file in the browser, and it will get com.apple.quarantine and com.apple.metadata:kMDItemWhereFroms). These are also shoveled into AppleDouble files. Actually, resource fork is just com.apple.ResourceFork extended attribute.

I'm not sure that there are more than 1% of mac users do not exchange files with other people, possibly on other platforms. What do they use the computer for, then? iPad would be more suitable then.

Re:APFS is not safe to use with names which have Unicode nor... by mattnewton on Tuesday June 06, 2017 @11:56AM (#14498176)
GP is saying they don't pass around flash drives to do so, not that they don't share files.
Re:APFS is not safe to use with names which have Unicode nor... by klodolph on Tuesday June 06, 2017 @03:24PM (#14500361)
I think that there are well more than 1% of users, maybe 20% or higher, that never share files except to attach a picture to an email, upload it to Facebook, or something similar like that. It's a bit of a nitpick, though, but we do often forget the "non-power" computer users.