APFS does not normalize Unicode filenames

Re:APFS does not normalize Unicode filenames by vbezhenar on Saturday March 25, 2017 @07:08AM (#13955025)

I'm not sure if normalization is good idea (generally because Unicode is complex beast and moving that complexity inside a kernel should be carefully weighted), but I'm sure that it doesn't solve any real problem. Characters "A" and "А" looks identical, unless you're missing Cyrillic font, but they won't be normalized, because they are completely different characters. There are many more other visually identical strings. So while normalization might solve some simple problems, it's not a complete solution, so filesystem might just treat names as byte arrays and let user solve his problems.

[ Parent | on HN ]

Re:APFS does not normalize Unicode filenames by TazeTSchnitzel on Saturday March 25, 2017 @10:26AM (#13955794)

Confusable characters look similar or the same to humans.

Canonically equivalent Unicode sequences look the same to machines.

The latter is a much more significant problem, because it can wreak havoc with interoperability.

[ Parent | on HN ]

Re:APFS does not normalize Unicode filenames by acdha on Saturday March 25, 2017 @10:46AM (#13955901)

Very true - and it's amplified by inconsistency allowing problems to spread further before being noticed. At work I deal with a lot of Bag-It archives where we have a text manifest of checksums accompanying files on disk and this reliably bites users of tools when something (archive or network transfer tool, Git or SVN, transition to/from a Mac with HFS+, etc.) causes the encoding in the manifest not to match the local filesystem, and the confusion is amplified because some tools will handle normalization differences so the bug report is “why does tool A say this file is missing when Explorer/Finder and tool B say it's fine?”

[ Parent | on HN ]

Re:APFS does not normalize Unicode filenames by slrz on Saturday March 25, 2017 @09:54PM (#13958591)

Canonically equivalent Unicode sequences look the same to machines.

Memcmp disagrees, as do the default equality operators of most programming languages in existence.

[ Parent | on HN ]

Re:APFS does not normalize Unicode filenames by TazeTSchnitzel on Saturday March 25, 2017 @10:04PM (#13958641)

Sure, but normalisation can nonetheless happen automatically and implicitly in many places.

[ Parent | on HN ]

Re:APFS does not normalize Unicode filenames by vbezhenar on Sunday March 26, 2017 @03:59AM (#13959711)

Rust uses separate string type for file names. I think, that's a good approach. If language normalizes strings behind your back, that's not very good.

[ Parent | on HN ]