Many unicode languages work with UTF-8 or UTF-16 internally. So working with the "transfer format" is common practice.
While it may not be necessary to know who the languages your program in work under the hood, expert programmers do want/need to know. That way they can write better code, or switch to another language or get the language devs to improve their internal handling.
The programmer shouldn't have to know that a newline character is written as \n in a JSON string.
The JSON string "a\nb" take 6 characters to write, but it's length should be given as 3.
99% people want to manipulate a JSON model, not the JSON (or BSON) serialization itself. The 1% can still use a byte array and do whatever hacks they like.
A better example is if you want to find a newline in a string. If you do a find it in a UTF-16 string it may be position 8 and a find in UTF-8 may be position 12. Does it matter what the actual number is? NO. You just pass it to the next function or whatever.
I'd call it a slight improvement. And after the major step back of using 2 to 4 times more memory for strings I'd call it a net loss.
> A code point string has other niceties, like being indexed by simple integers.
Again, no benefit of this. The only argument I've heard here is to prevent bad slicing and I've shown a way to prevent that.
> Also, we mustn't lose sight of what the T means in UTF-8: transfer. It's not called UPF-8 (the Unicode processing format in 8 bits).
By this argument we can't use UTF-16 or UTF-32 for internal processing of strings either. Back to code pages then.