News for hackers. Stuff that matters.
Re:APFS is not safe to use with names which have Unicode nor... by nottorp on Tuesday June 06, 2017 @07:10AM (#14496330)
To be honest, I blame Unicode. Why allow different representations for the same character, and then provide a normalized form anyway, except it's not one normalized form but several? Sounds like job security to me.
Re:APFS is not safe to use with names which have Unicode nor... by the_mitsuhiko on Tuesday June 06, 2017 @08:51AM (#14496804)
> To be honest, I blame Unicode. Why allow different representations for the same character, and then provide a normalized form anyway, except it's not one normalized form but several?

That's not really the issue here. Even if you can avoid normalization in some languages the problem will come back in others. For instance what are you going to do about invisible characters, control characters or more? Traditionally software attempted to just ignore the problems and let it blow up in other places. For some fun issues just try whitespace and bell characters in filenames and navigate around in your shell.

Normalization is just one of many issues with filenames.

Re:APFS is not safe to use with names which have Unicode nor... by Piskvorrr on Tuesday June 06, 2017 @07:22AM (#14496389)
Blame the encodings that came before it: Latin-1 and Windows-1252, and all the other ones. Hindsight is always 20/20, y'know.
Re:APFS is not safe to use with names which have Unicode nor... by jrimbault on Tuesday June 06, 2017 @07:47AM (#14496492)
I'm just being sarcastic: should we also blame the computers of old for not being able to handle more than 255 characters ?
Re:APFS is not safe to use with names which have Unicode nor... by cryptonector on Tuesday June 06, 2017 @04:48PM (#14501195)
No, just developers. Of course, the use of bytes for characters goes back a long time, to times when computers had small memories and disk (and other) storage capacities. And to even before then, to the days of telexes and typewriters. It's completely understandable. But UTF-8 is genius, which is why we use it.

Incidentally, ASCII was actually a multi-byte codeset... since one could combine most lower-case characters with BS (backspace) and overstrike with apostrophe, backtick, tilde, comma (for cedilles), and double-quotes (for umlauts), or with the same char for bold, or with underscore for underline. nroff(1) still uses this for bold and underscore, no?

Re:APFS is not safe to use with names which have Unicode nor... by thaumasiotes on Wednesday June 07, 2017 @02:15AM (#14504094)
I've always wondered what the concept of a "backspace" character was supposed to mean. (I tried printing it to erase a previously printed character, but that doesn't work.)

I guess it's another relic of the idea that computer output goes to a printer rather than a display?

Re:APFS is not safe to use with names which have Unicode nor... by cryptonector on Wednesday June 07, 2017 @03:28PM (#14509201)
Yes! Terminal vendors had to explicitly support this in terminals, though maybe not just because of printers but because it's actually quite useful.

Note that BS only produces overstrike when followed by certain characters (which we might term "combining" for the fun of it), while most will just change the character at that location. A tty spinner is just |BS/BS-BS\BS|BS... with some delay between each -- no overstrike there.

Re:APFS is not safe to use with names which have Unicode nor... by Piskvorrr on Wednesday June 07, 2017 @04:18AM (#14504524)
Indeed it is. Or rather, output goes to a remote printer. Morse code - teletype (backspace appears here) - mainframe - ASCII - Unicode (you are here).
Re:APFS is not safe to use with names which have Unicode nor... by Piskvorrr on Tuesday June 06, 2017 @09:11AM (#14496908)
Not really. In a barely-networked and severely resource-constrained world (kilobytes of memory, not gigabytes), this decision made sense. Shift happens.
Re:APFS is not safe to use with names which have Unicode nor... by nottorp on Tuesday June 06, 2017 @07:53AM (#14496518)
I don't remember those having multiple representations for the same character in the same encoding or normal forms...
Re:APFS is not safe to use with names which have Unicode nor... by Piskvorrr on Tuesday June 06, 2017 @09:02AM (#14496863)
Indeed not: there is an almost infinite number of things that Latin-1 did not have; e.g. any characters beyond Western Europe. While keeping the 128-256 block compatible was a part of early Unicode (hence the "one-glyph" é), having composed characters was a Unicode primary design goal (hence e and the composing accent). A pure Unicode implementation would have been better, maybe; what we have instead is one that has gained mass adoption.

https://en.wikipedia.org/wiki/Latin-1_Supplement_(Unicode_bl...

Re:APFS is not safe to use with names which have Unicode nor... by rurban on Tuesday June 06, 2017 @11:57AM (#14498190)
The biggest actual problem is Greek vs Cyrillic. They appear the same, but use different encodings. Any filename (directory entry as identifier) must forbid mixed scripts. See TR31 http://www.unicode.org/reports/tr31/ Combining marks or RTL switching tricks are the other popular spoofs.
Re:APFS is not safe to use with names which have Unicode nor... by cryptonector on Tuesday June 06, 2017 @05:21PM (#14501457)
Confusables are a real problem.

Some Cyrillic and Greek letters even look like Latin letters. And in some fonts '1' (one) and 'l' (ell) look the same even in Latin.

Dealing with these is much harder than with normalization forms. And confusables are a source of serious security headaches.

Re:APFS is not safe to use with names which have Unicode nor... by Piskvorrr on Wednesday June 07, 2017 @04:56AM (#14504663)
1Il|!
Re:APFS is not safe to use with names which have Unicode nor... by kps on Tuesday June 06, 2017 @11:54PM (#14503626)

  > Any filename (directory entry as identifier) must forbid mixed scripts.
So I'm not allowed to name my files “A=πr²” or “10kΩ Резисторы”?
Re:APFS is not safe to use with names which have Unicode nor... by rurban on Wednesday June 07, 2017 @08:35AM (#14505776)
According to me not if you mixed Greek with Cyrillic, and not Math with Cyrillic. Unfortunately nobody cares about Unicode identifier security models. Garbage out is the most popular ideology.
Re:APFS is not safe to use with names which have Unicode nor... by kps on Wednesday June 07, 2017 @11:34AM (#14507255)
I take the view that modern file names are human readable labels, not access keys.

“10kΩ Резисторы” is Cyrillic, Latin, and Greek (U+2126 canonically maps to U+03A9). A proposal that disallows that but allows “10kΩ Resistors” is a political non-starter.

Re:APFS is not safe to use with names which have Unicode nor... by rurban on Wednesday June 07, 2017 @05:45PM (#14510107)
The Unicode consortium has published recommendations with various security levels everybody enjoys to ignore. Following even the weakest level would be a starter, but not in current filesystems community. This needs to start in a new OS or with high profile spoofing attacks, such as on GitHub.
Re:APFS is not safe to use with names which have Unicode nor... by nottorp on Tuesday June 06, 2017 @09:22AM (#14496983)
Well, composed characters may or may not have been a good idea, although they're a pain for text search if you ask me.

But why oh why did they have to specify FOUR normalization forms?

Ref: http://www.unicode.org/reports/tr15/#Norm_Forms

Re:APFS is not safe to use with names which have Unicode nor... by cryptonector on Tuesday June 06, 2017 @04:43PM (#14501146)
Pre-composed characters were needed to make transcoding to ISO-8859-* fast.

Also, for Hangul, though decomposition is the better form to use normally (since it's phonetic, not syllabic), conversion to pre-composed syllabic form is very useful sometimes.

Even without pre-composition there would have been multiple equivalent forms of writing any character that requires more than one combining codepoint.

The only way to have avoided normalization altogether would have been to have no combining codepoints, and only a complete set of pre-compositions. This would have been less flexible, and very obnoxious for Hangul and possibly others.

I believe the need for normalization was simply unavoidable. It's not the fault of Unicode but the fault of humans' script designs. And it's OK.

Re:APFS is not safe to use with names which have Unicode nor... by the_mitsuhiko on Tuesday June 06, 2017 @10:48AM (#14497617)
> But why oh why did they have to specify FOUR normalization forms?

Ignoring compatibility normalizations there are really only two and neither of those two you can remove. So that's a bit of an odd question. If you want to remove the compatibility forms that's fair but nobody is force to use them. You can consider them "partially normalized" or "not normalized at all" for all intents and purposes

Re:APFS is not safe to use with names which have Unicode nor... by cryptonector on Tuesday June 06, 2017 @04:44PM (#14501153)
You can ignore the K forms. So that's two forms, really, NFC and NFD.
Re:APFS is not safe to use with names which have Unicode nor... by TazeTSchnitzel on Tuesday June 06, 2017 @10:23AM (#14497401)
Because you need a normalisation form fit for your particular purpose.
Re:APFS is not safe to use with names which have Unicode nor... by Piskvorrr on Tuesday June 06, 2017 @09:24AM (#14496992)
¯\_(ツ)_/¯
Re:APFS is not safe to use with names which have Unicode nor... by the_mitsuhiko on Tuesday June 06, 2017 @10:36AM (#14497508)
> I don't remember those having multiple representations for the same character in the same encoding or normal forms...

But that's not really what is happing here. What is happening is that the renderer decides to merge some code points into one glyph. That was true long before unicode as well (for instance ligatures) or even on old terminals with overstriking. Unicode just made this more explicit.

Re:APFS is not safe to use with names which have Unicode nor... by TazeTSchnitzel on Tuesday June 06, 2017 @10:17AM (#14497357)
Compatibility and efficiency are two big reasons why you would want to have composed forms like e.g. the single codepoint for é: legacy encodings also have a single codepoint for it, and it's more efficient to represent it with one codepoint than with two.

So, we need composed characters, but surely combining characters aren't needed, then? Well, no, because it's unreasonable to include a codepoint for every single combination, especially once you have multiple diacritics on a single character, for example. Also, compatibility factors in here too, because legacy encodings likewise have combining characters.

Re:APFS is not safe to use with names which have Unicode nor... by cryptonector on Tuesday June 06, 2017 @05:28PM (#14501523)
And also: scripts evolve. People may well start combining various diacritical marks in ways not done before.

Also, it can be useful to decompose in some cases. For example, it's useful for removing diacritical marks, which can be useful for fuzzy searches. Hangul is really phonetic, not syllabic, so there it's the reverse: decomposed forms are most useful, except when you need to think in syllables.

Re:APFS is not safe to use with names which have Unicode nor... by Ono-Sendai on Tuesday June 06, 2017 @10:39AM (#14497538)
Combining characters seems like a pretty stupid thing to have in a character set/encoding. Does anyone know how many combined characters are actually needed?
Re:APFS is not safe to use with names which have Unicode nor... by cryptonector on Tuesday June 06, 2017 @05:26PM (#14501508)
More than there are pre-composed codepoint assignments for.
Re:APFS is not safe to use with names which have Unicode nor... by cryptonector on Tuesday June 06, 2017 @01:37PM (#14499204)
It's not Unicode's fault. Unicode took a terrible situation (lots of complex scripts) and did the best it could to give us a unified way of representing text in all scripts.
Re:APFS is not safe to use with names which have Unicode nor... by userbinator on Tuesday June 06, 2017 @07:22AM (#14496391)
More like a typical outcome of design-by-committee.