Re:UCS vs. UTF-8 as Internal String Encoding
by masklinn on Thursday May 28, 2015 @01:04PM
(#9619502)
> Both of these things are necessary, in the end, for the controller backing any text-editing control, right?
A controller backing a text-editing control can easily maintain an intermediate data structure mapping code points (or even code units) to whatever its unit of editing is.
> And, philosophically, I'd say that something close to (but not quite) this, is what people want to know from a lot of systems when they ask for a string's "length." They don't care how much memory the string is taking up (though they can have some low-level assertions to that effect); they care about whether the supplied text will fit in a given em-width container, or a given number of form blanks. This is what Twitter actually cares about these days; this is what wc(1) or Microsoft Word wants to tell you when you look at "character count"
Sure, I don't disagree about that.
> Thus why I said "glyph"—people expect "ss" to be two glyphs; "fi" to be two glyphs, and "ß" to be one glyph.
Except fi is a single glyph (but should be two grapheme clusters).
> Which is all to say, maybe raw Unicode strings are good for systems languages—but in my day-to-day programming solving business-domain problems, I'd much rather be using a "text library" that provides "text objects" with a length measured in glyphs.
That's a completely different concern, the internal representation and external interface to the string don't have to match. And in my experience, neither glyph length nor grapheme cluster length are of much use outside of providing a mostly pointless character count for a document. Useful metrics are encoded size (for storage) and rendered dimensions (to fit things to displays), the latter not being computable without more or less fully rendering the string anyway. Neither would benefit much or at all from a string representation with first-class grapheme clusters (let alone first-class glyphs)
A controller backing a text-editing control can easily maintain an intermediate data structure mapping code points (or even code units) to whatever its unit of editing is.
> And, philosophically, I'd say that something close to (but not quite) this, is what people want to know from a lot of systems when they ask for a string's "length." They don't care how much memory the string is taking up (though they can have some low-level assertions to that effect); they care about whether the supplied text will fit in a given em-width container, or a given number of form blanks. This is what Twitter actually cares about these days; this is what wc(1) or Microsoft Word wants to tell you when you look at "character count"
Sure, I don't disagree about that.
> Thus why I said "glyph"—people expect "ss" to be two glyphs; "fi" to be two glyphs, and "ß" to be one glyph.
Except fi is a single glyph (but should be two grapheme clusters).
> Which is all to say, maybe raw Unicode strings are good for systems languages—but in my day-to-day programming solving business-domain problems, I'd much rather be using a "text library" that provides "text objects" with a length measured in glyphs.
That's a completely different concern, the internal representation and external interface to the string don't have to match. And in my experience, neither glyph length nor grapheme cluster length are of much use outside of providing a mostly pointless character count for a document. Useful metrics are encoded size (for storage) and rendered dimensions (to fit things to displays), the latter not being computable without more or less fully rendering the string anyway. Neither would benefit much or at all from a string representation with first-class grapheme clusters (let alone first-class glyphs)