News for hackers. Stuff that matters.
Re:Why to normalize Unicode strings by jrochkind1 on Wednesday March 13, 2019 @01:05PM (#19380827)
The official unicode documentation on normalization is good reading, and quite readable. It's actually an even more complicated topic than OP reveals, but the Unicode Standard Annex #15 explains it well.

http://unicode.org/reports/tr15/

OP has a significant error:

> You can choose whatever form you’d like, as long as you’re consistent, so the same input always leads to the same result.

Not so much! Do _not_ use the "Compatibility" (rather than "Canonical") normalization forms unless you know what you are doing! UAX15 will explain why, but they are "lossy". In general, NFC is the one to use as a default.

Re:Why to normalize Unicode strings by SloopJon on Wednesday March 13, 2019 @02:12PM (#19381648)
Ligatures, which are mentioned in the article, are a good example of the distinction between canonical and compatibility forms.

If your input contains "traffic", you don't necessarily want NFC to insert the "ffi" ligature and turn it into "tra\ufb03c". The ligature is generally a poor choice in a monospaced font, for example.

Similarly, if you've gone to the trouble to insert the ligature, you don't necessarily want NFD to strip it out. However, with NFKC or NFKD, a search for "traffic" will find the string.

Re:Why to normalize Unicode strings by lelf on Wednesday March 13, 2019 @04:48PM (#19383595)
Ligatures are not a good example actually. They are obsolete by Unicode, i. e. there's no way to turn ffi⟨normal⟩ into ffi⟨ligature⟩. (But NFKD(ffi⟨ligarure⟩)≡ffi⟨normal⟩ of course.)
Re:Why to normalize Unicode strings by jrochkind1 on Wednesday March 13, 2019 @05:04PM (#19383769)
You're right that there's no way to apply any specified unicode normalization to turn `ffi`(normal) into `ffi`(ligature).

But they're a good example of the difference between "canonical" and "compatible" normalization anyway, in the other direction. Which is really the only direction that matters to illustrate the difference anyway.

NFKD (or NFKC too I think?) will, as you say, turn the ligatures into their "individual" forms. It's different glyphs, and there is at that point no way to know which it was "originally", and no way to convert it in the other direction. The "compatibility" normalizations are "lossy".

The "canonical" normalizations on the other hand are basically 'lossless' with regard to "glpyhs". Unless you actually _cared_ that it was represneted with a combining diacritic before, which in 99.9% of cases you don't. You should have exactly the same symbols on the screen after a 'canonical' normalization. And for any x, NFC(x) == NFC(NFD(NFC(x)).

The compatibility normalizations are super useful, because as the person up there mentioned, you often want a search query for `ffi` to match on `ffi` (and vice versa). But they are intended to lose symbolic representation (ffi and ffi are now the same thing with no way to distinguish), where the canonical normalizations are not.

I wouldn't say the `ffi` ligature is "obsoleted" in any way. People still use it all the time. We both just included it in our comments, unicode was happy to support that. Unicode is so happy for you to use it, that it provides compatibility normalization to make it _easier_ to recognize that it means the same thing as "ffi". :)

Re:Why to normalize Unicode strings by zokier on Wednesday March 13, 2019 @01:43PM (#19381311)
> In general, NFC is the one to use as a default.

Why nfc instead of nfd?

Re:Why to normalize Unicode strings by jrochkind1 on Wednesday March 13, 2019 @02:10PM (#19381625)
Not really sure why NFC has become the general standard.

UAX#15 notes that:

> The W3C Character Model for the World Wide Web 1.0: Normalization [CharNorm] and other W3C Specifications (such as XML 1.0 5th Edition) recommend using Normalization Form C for all content, because this form avoids potential interoperability problems arising from the use of canonically equivalent, yet different, character sequences in document formats on the Web. See the W3C Character Model for the Word Wide Web: String Matching and Searching [CharMatch] for more background.

One of the W3C documents cited says:

> NFC has the advantage that almost all legacy data (if transcoded trivially, one-to-one, to a Unicode encoding), as well as data created by current software or entered by users on most (but not all) keyboards, is already in this form. NFC also has a slight compactness advantage and is a better match to user expectations in most languages with respect to the relationship between characters and graphemes.

https://www.w3.org/TR/charmod-norm/

Re:Why to normalize Unicode strings by jfk13 on Wednesday March 13, 2019 @02:11PM (#19381638)
To conform with W3C recommendations, for one thing:

"Content authors SHOULD use Unicode Normalization Form C (NFC) wherever possible for content."[1]

It's a SHOULD rather than a MUST. Non-NFC content is not a spec violation, but using NFC is strongly advised.

[1] https://www.w3.org/TR/charmod-norm/#normalizationChoice

Re:Why to normalize Unicode strings by minitech on Wednesday March 13, 2019 @01:53PM (#19381443)
Saves space, is just what people expect, easier to work fast and loose with when you want to do that.
Re:Why to normalize Unicode strings by rurban on Thursday March 14, 2019 @01:35AM (#19386745)
But needs a third more time than NFD. Apple choose the long form NFD in its old filesystem. NFC needs three steps, NFD only the first two. The size rarely matters that much.