Perl 6 has long specified an interesting and rich approach to Unicode. Have you explored it?
(An initial implementation for an initial release of the Rakudo compiler was worked on over the last few years and declared done, modulo bugs, a few weeks ago. A recent blog post provides a friendly overview.[1].)
One notable difference is that Swift adopts an iteration-only view of strings for dealing with characters whereas Perl 6 does not. One consequence is that indexing operations in Perl 6 are O(1) (time) at the expense of potentially (and for the current implementation, typically) greater use of RAM.
> BTW to your point on len(x) + len(y). The answer is 2 if you define len() on Unicode scalars but 1 if you define it on characters. Why? because len("\u301") should be 0.
Are you 100% sure of this of your last sentence ("should be 0")? I get that it's intuitively reasonable that an isolated base modifier be length 0 but I thought the Unicode standard specified (or at least recommended) that it be 1, and I note that Perl 6 treats isolated base modifiers (or at least this particular one) as length 1:
> ("a").chars 1
> (0x301.chr).chars 1
> ("a" ~ 0x301.chr).chars 1
So both Swift and Perl 6 correctly report the length of "a" as 1 and an "a" concatenated with "\u301" as 1 but disagree on what to do with an isolated base modifier.
> It is, of course, true that getting this right is likely to be substantially more expensive than getting it wrong but that doesn't mean it can't be done.
Indeed.
[1] https://perl6advent.wordpress.com/2015/12/07/day-7-unicode-p...
You learnt ASCII. Then you learnt about escaping to get around limitations in ASCII. Then you learnt that Unicode got around non-Latin issues with ASCII.
Each step has added cognitive load. It seems surprising to me that the next step wouldn't.
I'm not saying that the string API in Python is perfection, but it had a specific role to fill in the larger Python ecosystem, and it does that very well.
Anyway... I would say that len(x) == 0 should be the same as saying that x is empty. According to the Unicode standard, the number of grapheme clusters in "\u0301" is 1, anyway! And on most systems, the string will display as an acute accent in its own space, and you can place the cursor before or after it.
Not like the set of valid "cursor positions" in a string isn't a locale-dependent value, anyway. Grapheme clusters can be thought of to be locale-independent but they are NOT the same as cursor positions.
If I recall correctly the modifier on it's own isn't correct. There is a base that should go before it to produce a valid character. I don't recall the base unfortunately and I don't have the spec to hand.
I mean this from the standard: "Nonspacing combining marks used by the Unicode Standard may be exhibited in apparent isolation by applying them to U+00A0 no-break space. "
You can have valid Unicode that doesn't do this. But it's not intended to represent the character as some kind of textual thing. It's intended so that you can build up Unicode strings sensibly. Same idea as allowing mis-matched surrogates.
But that's a choice and not the only one. Sure you're likely to implement your Unicode string as an array of some type. But that doesn't mean that the only sensible approach is to expose that array directly to the user. Or more specifically as the primary interface.
For my money, Swift has probably the most comprehensive Unicode string API I've seen [0]. What they do is, essentially, have an opaque String type but support various "views" as properties. The main view is called .characters and represents a collection of grapheme clusters. They also have properties that present Unicode scalars, utf8 encoding etc.
Their API is complex, no question. But then proper handling of Unicode is complex. But it does show that there are other options than simply exposing the Unicode scalars.
BTW to your point on len(x) + len(y). The answer is 2 if you define len() on Unicode scalars but 1 if you define it on characters. Why? because len("\u301") should be 0. It's not a character, it's a base modifier. It is, of course, true that getting this right is likely to be substantially more expensive than getting it wrong but that doesn't mean it can't be done.
[0] https://developer.apple.com/library/ios/documentation/Swift/...