Which wide characters are you talking about? Because on Windows, where wide characters are 16 bits, it's quite possible to get corrupted strings (and in fact quite a few well-known programs, written by quite well-known software companies, make this exact mistake).

All you need to do is index/slice a string half-way through any character that is outside Unicode's Basic Multilingual Plane

This drives me crazy. The Win32 API was designed for UCS-2. Then UTF-16 came out and the API was shoehorned to use it but as you said they still haven't caught all the places where it still thinks it's UCS-2.

"wide enough" So if it's Unicode, each element is a Unicode character.

First of all, Unicode doesn't define characters it defines codepoints.

I get that this might seem pedantic, but it's important to be pedantic about this, otherwise misconceptions and ambiguities occur e.g. 'just use wide characters' - the definition of which changes depending on the platform.

Second of all, "wide enough" for all intents and purposes means 32 bits. Technically Unicode only needs 21 bits to cover the currently defined codespace, but computers don't deal well with that and so 32bits is the minimum "wide enough" character size.

This creates a lot of wasted space and memory, not to mention pushes medium length strings across cache line boundaries for very little benefit - the ability to directly index/slice strings without accidentally corrupting data.

Now obviously you want to avoid accidentally corrupting data, the tradeoff comes down to whether you need direct, arbitrary indexing, or if it's worth doing some processing to determine the correct place to split in order to make space gains.

The technical world has come down overwhelmingly in favour of the latter, and that's why you see hardly anyone using utf-32. It's simply not as good a solution for most real world concerns.

Not impossible. If you slice in the middle of a grapheme then you get a corrupted string as well. You'll get an alternate glyph instead of a square box but it's still corrupted.

It's not a corrupted string. You may have mangled it in a way that doesn't preserve all semantics, but no one will crash with an encoding issue.

A "string" means "a sequence of characters". Wide characters (or the equivalent interface) preserve this property.

Graphemes operate at a higher level than characters. You could construct a grapheme-strings, I suppose, but that has tons of edge cases, and if you don't like character-strings, I doubt you will like grapheme-strings.

Why should it crash? The proper procedure when validating a UTF-8 string is to replace errors with U+FFFD.

The term character has many meanings. Graphemes are characters and that's what most users expect, something that's displayed as a single graphical unit.

I use "character" in the same way that the Unicode Consortium uses the word. Though "code point" would be more precise.

That's what they were hoping for. Didn't tuen out thst way. From icu-project.org:

"As with glyphs, there is no one-to-one relationship between characters and code points. What an end-user thinks of as a single character (grapheme) may in fact be represented by multiple code points; conversely, a single code point may correspond to multiple characters."

Or, if by "string" we mean "sequence of code points" (rather than graphemes) then it doesn't get corrupted by any chopping or rearrangement which only permutes the code points.

If we chop UTF-8, we can end up with bad characters, or possibly invalid overlong forms.

I consider changing a glyph to some other glyph(s) as corruption. Take an emoji flag character as an example. Split it between the code points and you end up with two boxed letters.

If you chop in the middel of a code unit then you end up with U-FFFDs. In both cases the visual representation has been altered.

As I wrote elsewhere it is easy for the slice routines of a language to check to see if the programmer tried to slice in the middle of a code unit and either return an error or just advance to the start of a code unit.

UTF-8 slicing destroys characters and graphemes.

Slicing a code-point-character string destroys only graphemes.

Clear win.

A code point string has other niceties, like being indexed by simple integers. If end is the index of the last code point of a grapheme, then the next grapheme starts at end + 1.

If end is the index of the last UTF-8 encoding of a code point, then the next grapheme does not start at end + 1.

We can have it so that it does by making end point to the last byte of the UTF-8 encoding of the code point; but then it doesn't point at the start of the character, recovering which is awkward.

The code uglification can be addressed by piling on abstractions: integer-like iteration gizmos that can be incremented and decremented thanks to function or operator overloading.

I feel that that level of abstraction has no place in character-level data processing, if anywhere, whose basic operations should be expressible tersely in a few machine instructions.

Also, we mustn't lose sight of what the T means in UTF-8: transfer. It's not called UPF-8 (the Unicode processing format in 8 bits).

Working with UTF-8 instead of with the objects that UTF-8 denotes is like working with a textual representation of Lisp s-expressions that still contain the parentheses and whitespace delimitation, and quotes around strings and so on, refusing to parse them to obtain the object which they represent. People who do this should immediately turn in their CS degrees.

All those other issues you refer to are addressed by more parsing. If you want the glyphs, the correct thing is to parse the code-point string and make a list or vector of glyph representations.

With that representation you can still break the text "carpet" into "car" "pet" which destroys semantics; that is dealt with by parsing into words.

Chopping lists of words destroys phrases; so parse phrases, and transform at the phrase level.

And so on.

> Clear win.

I'd call it a slight improvement. And after the major step back of using 2 to 4 times more memory for strings I'd call it a net loss.

> A code point string has other niceties, like being indexed by simple integers.

Again, no benefit of this. The only argument I've heard here is to prevent bad slicing and I've shown a way to prevent that.

> Also, we mustn't lose sight of what the T means in UTF-8: transfer. It's not called UPF-8 (the Unicode processing format in 8 bits).

By this argument we can't use UTF-16 or UTF-32 for internal processing of strings either. Back to code pages then.

His point was that the transfer format should be conceptually independent of the processing. Obviously it has to be encoded in RAM in some way, but the programmer doesn't need to worry about the memory layout.

Why? Why should it be conceptually different if it's easy to work with the encoded form?

Many unicode languages work with UTF-8 or UTF-16 internally. So working with the "transfer format" is common practice.

While it may not be necessary to know who the languages your program in work under the hood, expert programmers do want/need to know. That way they can write better code, or switch to another language or get the language devs to improve their internal handling.

Think of JSON.

The programmer shouldn't have to know that a newline character is written as \n in a JSON string.

The JSON string "a\nb" take 6 characters to write, but it's length should be given as 3.

99% people want to manipulate a JSON model, not the JSON (or BSON) serialization itself. The 1% can still use a byte array and do whatever hacks they like.

Bad example. If you want to embed that string in your code you have to type those 6 characters anyways.

A better example is if you want to find a newline in a string. If you do a find it in a UTF-16 string it may be position 8 and a find in UTF-8 may be position 12. Does it matter what the actual number is? NO. You just pass it to the next function or whatever.