The manifesto is not in any way advocating that you use byte length as a substitute for string length. (It does argue that string length is not very commonly necessary, and has unclear semantics because of the multiple different definitions for "character", but these arguments are unrelated to the statement you quoted.)
Here's the point the author was making, which you missed. Take a non-BMP code point like 'š', which is U+1D11E. In UTF-16 this is represented by code units D834 and DD1E. If you try to use a "substring" operation to take the first "character" of this with a C#/.NET substring operation, you will get an invalid string, since D834 by itself is not a valid UTF-16 string.
> I want to work on top of an abstraction which lets me treat text as text.
If you think that UTF-16 will let you say string[i] and always get the i'th character, you are mistaken. That is one of the main points of the essay.
> Yes, their are cases where the abstraction will leak. But those cases are very far and few in between.
If you write your apps this way, then you don't really support Unicode, you just support the BMP, without combining characters.
In addition, while .NET methods will work correctly for "ĆĆĆ ", they have incorrect behavior when it comes to any characters that lie outside the BMP, which includes many CJK characters. So, .NET only meets your requirements if you never interact with such languages.
...and saying that, it'd still be a good idea.
Everything works on "logical characters" - arbitrary vectors of codepoints, as you say. There's still a number of edge cases I have yet to work out as to what exactly is considered a character, though. (I just added support for a single code point encoding multiple characters, for example.)
I'm not so sure that making thing reliant on a font would be the best way to solve that, though. I'd intuitively say that there should be less coupling between rendering choices and internal encoding than that.
The twist is that each node in the rope can only store characters of the same physical length in bytes (and same number of logical characters per physical character). This means that in the typical case (most characters require the same number of bytes to encode) it doesn't add too too much overhead. Still not something I would consider as the base String type for a lower-level language, though.
There are a few simple optimizations that I have yet to do (encode smaller characters as what would be ordinarily be invalid longer encodings, if it makes sense (a single one-byte character in the middle of a bunch of two-byte characters, for example), that sort of thing.)
It seems to work fairly well, so far. Or at least it tends to give "common-sensical" results, and avoids a large chunk of worst-case behavior that standard "prettified character array" strings have.
If we were serious about a character-oriented API, we definitely wouldn't want to introduce character rendering rules into places like the kernel. But I don't think we'd necessarily have to.
The best solution, I think, would be to decompose fonts into two pieces:
1. a character map (a mapping from paramaterized codepoint sequences to single entities known to the font),
and 2. a graphemes file (the way to actually draw each character.)
The graphemes file would be what people would continue to think of as "the font." And the graphemes file would specify the character map it uses, in much the same way an XML/SGML document specifies a DTD.
As with DTDs, the text library built into the OS would have a well-known core set of character maps built in, and allow others to be retrieved and cached when referenced by URL. The core set would become treated something like root CAs or timezone data are now: bundled artifacts that get updated pretty frequently by the package manager.
From that section they explain that for text manipulation [e.g: cursor position & manipulation of text under the cursor] the programmer should be counting grapheme clusters; whereas for storage [memory & disk] concerns the programmer _should_ care about the number of codepoints.
They go on to say that counting the number of characters is up to the rendering engine and is completely unrelated to the number of codepoints.
I don't think the _manifesto itself_ is wildly misguided. In that section they were merely pointing out _how .NET and Java currently report string length._
At least as I read the manifesto: they seem to believe that counting codepoints is useful _but orthogonal_ to counting characters.
Regardless of the encoding you choose, you need to understand how it represents characters.
Also FYI, most programs don't display text on the screen. To most programs, text is a sequence of bytes to be shuffled around, and nothing more. Display and text manipulation is the minority case.
The others would be good reasons though.
Both C# and Java offer a 16 bit char type, which is less than a Unicode character, congratulations. The .NET indexer str[i] works in units of the internal representation, hence a leaky abstraction once again. Substring methods will happily return an invalid string, cutting a non-BMP character in parts.
While theoretically true, for most practical purposes, this reeks of a USA/American/English bias and lack of real world experience.
You know what? I want to know that the text "ĆĆĆ " is three characters long. I dont want to know that it's a 6-byte array once encoded to UTF8. Anywhere in my code telling me this is 6 characters is lying, not to mention a violation of numerous business-requirements.
When I work with text I want to work with text and never the byte-stream it will eventually be encoded to. I want to work on top of an abstraction which lets me treat text as text.
Yes, their are cases where the abstraction will leak. But those cases are very far and few in between. And in all cases where it doesn't, it offers me numerous advantages over the PHPesque, amateurish and incorrect approach of treating everything as a dumb byte-array.
It's not. It's text in my program. It's text rendered on your screen. It's just a byte-array when we send it over the wire, so stop trying to pretend text isn't text.
This manifesto is wildly misguided.