This doesn't matter much for (normalized) western European text, but if the language in question needs to use separate diacritical code points you'll likely end up with hanging accents in the like. Swift is the only language I know of that has grapheme clusters as the default unit of character, I'd love to see it in more places.

[1]: http://unicode.org/reports/tr29/#Grapheme_Cluster_Boundaries

Moving to grapheme cluster boundaries means that the algorithm may work incorrectly if you input a string of Unicode N+1 to an implementation that only supports Unicode N. It also makes the "increment character" function very complicated. In the UTF-8 version, this looks roughly like:

    char *advance(char *str) {
      uint8_t c = (uint8_t)*str;
      /* Count the number of leading 1's */
      int num1s = __builtin_clz(~c) - 24;
      if (num1s == 0) return str + 1;
      return str + num1s;
    }

    char *advance_grapheme(char *str) {
      while (true) {
        uint32_t codepoint = read_codepoint(str);
        str = advance(str);
        uint32_t nextCodepoint = read_codepoint(str);
        /* This is typically something like table[table2[codepoint >> 4] * 16 + codepoint & 15]; */
        GraphemeClusterBreak left = lookupProp(codepoint);
        GraphemeClusterBreak right = lookupProp(nextCodepoint);
        /* Several rules based on left versus right... */
      }
      return str;
    }

(Probably a bit "unfair" to "pounce" on an off-hand paranthetical like this, but I'm in a bit of a pedantic mood...)

This is not true for e.g. Haskell. In Haskell it's defined as [Char], i.e. a list of characters. (Of course the Haskell community is suffering from that decision, but that's another story.)

I'm not sure why strings would need to be a fundamendal type, though. Sure, they would probably be part of the standard library for almost all languages, but they don't need to be "magical" in the way most fundamental types (int, etc.) are.

I with mtviewdave that more complexity means this should really be in the std lib. Java's charAt(int i) is misleading at best.

My point is that I don't think throwing people into the deep end and expecting them to grok the codepoint/grapheme division before they get to use the language is likely to be productive. Defaulting to graphemes carries the advantage that if a programmer who doesn't know much about languages that require more Unicode finesse uses it purely intuitively, they'll get things right in a lot of cases. Using codepoints, on the other hand, makes it easy to put text through a grinder while doing relatively innocent things.

Unfortunately while we're discussing the finer points of using graphemes and codepoints in APIs a million supplementary characters were brutally eviscerated by code running on languages that haven't quite gotten past UCS-2 XD

Really, this is not dissimilar to "what do you mean, there are more letters than A to Z?" issue that plagued software written in US back before Unicode became dominant. The way we (my perspective on this is as a native speaker of a language with a non-Latin alphabet) have eventually solved it is by basically forcing Unicode onto those people. It broke their simple and convenient picture of the world, and replaced it with something much more complicated. But it was necessary.

My position is that letting programmers get away with a simplistic view of text processing (by allowing defaults that "mostly" work) is what creates those issues. So adjusting the abstractions such that they expose more of the underlying complexity is a good thing. People SHOULD believe that doing text processing the right way is hard, because it is.

But, Perl 6 definitely gets it more right than other implementations I've seen.

What i find more frustrating is how the documentation for many systems describes the basic unit of text as a character, without specifying whether a code point or grapheme is meant, and without leading people to an explanation of the difference. There is still a lot of software that processes unicode text incorrectly, not because it is difficult to do so, but because nobody told the developer how things should be done.

However... How does a language or library attempting to abstract this part (like swift might) deal with the other, unrelated annoying aspects of Unicode? Even if 1 glyph is 1 "char", and we normalize all the inputs, there is still, say ... Bi-di text.

PS: bit of trivia that people forget these days, Win32 has been supporting it for longer than android/iOS.

Also Elixir and Perl 6. For a bit more info see https://news.ycombinator.com/item?id=

I agree, however, that it's completely irrelevant whether the indexes correspond to code units (i.e. byte offsets in UTF-8) or whether they correspond to code points (how it works in Python currently), as long as we have some way to store, compare, and otherwise manipulate locations within a string.

Some Rust developers at one point proposed making string indexes their own (opaque) type, as you suggest, so that they couldn't be confused with integers used for other purposes. The extra complexity of such an API meant that this proposal was never really taken seriously, and it only prevents a small category of programming errors.

You might be interested in looking at some string APIs which are mostly without string indexing, like Haskell's Data.Text, which is one of the most well-designed string APIs ever made.

https://hackage.haskell.org/package/text-1.2.2.1/docs/Data-T...

As for Windows, my Windows apps use UTF-8 everywhere, and then convert to wchar_t at the last possible moment when interacting with the Windows API. I believe this is what UTF-8 Everywhere suggests.

That's why I really like dealing with UTF-8. As you said you can just index by byte instead of having to worry about code point boundries. This is because a code point will never match inside a larger code point.

So if I search for a one byte quote character it will never match the second, third or fourth bytes in a larger code point. Same with other control characters.

Works when searching for 2 or 3 byte code points as well.

In fact it looks like even standard indexes have been deprecated, in favor of Iterators over the string:

https://doc.rust-lang.org/stable/std/primitive.str.html#meth...

Iteration is better IMO.

I've been using references pretty effectively to get around some issues like this, though in other cases Rc is my only resort.

If you have some code up on github, I'd be happy to take a look and see if there's some other options that are less cumbersome.

If you want to split the string into "characters" you need to do it a the grapheme level (multiple code points) and for that you need to use a unicode library. But that adds overhead when you're just scanning.

A JSON or XML scanner does not need the added overhead of advancing by code point or grapheme.

For parsing, it often makes sense to iterate over code points or code units, since many languages are defined in terms of code points (and you can translate that to code units, for performance). XML 1.1, JavaScript, Haskell, etc... many languages are defined in terms of the underlying code points and their character classes in the Unicode standard. JSON and XML 1.0 are not everything.

For parsing it's easier to just scan for a byte sequence in UTF-8 because you know what you're looking for ahead of time. If you're looking for a matching quote, brace, etc. you just need to scan for a single byte in your text stream. Adding a smart iterator to the process that moves to the start of each code unit is not necessary and will slow things way down.

I just gave JSON and XML as examples and not an exhaustive list. If you know the code points you are scanning for it's way more effecient to scan for their code units. The state machine in a paraer will be operating at the byte level anyways.

I have yet to see a good example where processing/iterating by code point is the better choice (other than the grapheme code of the unicode library).

If you're curious, here's the V8 tokenizer header file:

https://github.com/v8/v8/blob/master/src/parsing/scanner.h

You can see that it works on an underlying UTF-16 code unit stream which is then composed into code points before tokenization. This extra step with UTF-16 is a quirk of JavaScript.

If you think that V8 shouldn't be processing by code point, feel free to explain that to them.

For languages that only allow non ascii in string literals a pure state machine would suffice.

Not sure why you're mentioning parsers. At that point you you're dealing with tokens.

As for UTF-16 it's an ugly hack that never should have existed in the first place. Unfortunately the unicode people had to fix their UCS-2 mistake.

Since Javascript is standardised to be either UCS-2 or UTF-16 it probably made sense to make the scanner use UTF-16.

ECMAScript source text is represented as a sequence of characters in the Unicode character encoding, version 3.0 or later. [...] ECMAScript source text is assumed to be a sequence of 16-bit code units for the purposes of this specification. [...] If an actual source text is encoded in a form other than 16-bit code units it must be processed as if it was first converted to UTF-16.

Really I think you are arguing against the notion of "default iteration" altogether. As you say, the right type of iteration is context dependent, and it ought to be made explicit.

[0]: https://blog.golang.org/strings

It's especially amusing because on Python 3 strings internally cache their utf-8 equivalent if it was used once.

https://github.com/python/cpython/blob/master/Include/unicod...

I think Python got unicode in the same era as Java, so it's understandable that Python 2 doesn't work like this. But if they are going to break the whole world for unicode, I also think it would have been better to do something like Go does (e.g. the rune library).

Or throw an invalid index exception if the top to bits are one if that makes more sense for the language you're using.

So what you're saying, is that it's very easy to get corrupted strings by anyone who doesn't have an understanding of utf-8 at the bit-level - which in my experience seems to be the majority of programmers.

And indexing by code points doesn't solve the problem either. The majority of programmers don't know what a grapheme is or how to collate or sort unicode strings.

My only gripe with your argument is that I don't think it's easy to avoid corrupted text in modern text processing - which is precisely why there are libraries for it because it's actually really easy to get it wrong - even if you know what you're doing.

When I was working on a project before Unicode we would switch our dev PCs to the other languages we supported. What a pain that was. Only issues we had was when a translated string was much longer than the screen space allocated to it. I belive Swedish was the main culprit. No problems with simplified and traditional Chinese as those were more compact. I have no sympathy for dev shops that can't get internationalization right. As with everything else in the corporate dev world management doesn't seem to want to hire/retain the more experienced programmers.

I think you have a gripe with my argument because you may be missing my point. If a high level language chooses to let a programmer index into a UTF-8 string at the byte level (for performance and other reasons) it's very easy for it to prevent the the programmer from slicing in the middle of a code unit.

The reason being is that the language function to slice a unicode string would either throw an exception or just advance to the next valid index. There wouldn't be a way for the programmer to slice a unicode string in the middle of a code unit.

I get your point, it just doesn't apply to many real world situations I've seen where you don't have the luxury of just using a higher level language or a library that takes care of all these things, or keeping programmers who don't understand what they are doing away from that sort of thing.

The most egregious example that I've personally seen was a developer working on a legacy Cobol banking program that needed Chinese support retro-fitted to it.

The app was originally only developed with ASCII in mind and so sliced through strings willy-nilly, which naturally caused problems with Chinese text.

The developer working on the "fix" before me, was calling out to ICU through the C API of the version of Cobol that we used and was still messing things up - he'd actually modified ICU in some custom way to prevent the bug from crashing the program, but was still causing corrupted text.

I basically undid all his changes, and wrapped all COBOL string splicing to call a function that always split a string at a valid position - truncating invalid bytes at the start/end as necessary. Much simpler and resulted in the removal of an unnecessary dependency on ICU.

This bug had been outstanding for several months when I first joined that company, and it was the first one I was assigned to work on - and luckily for them they'd accidentally hired someone who had done lots of multilingual programming before.

it's very easy for it to prevent the the programmer from slicing in the middle of a code unit.

Okay, but even you made a mistake in your first example of what to do, and that's the sort of code that someone who knows what they are doing could write, and will seem to work in the conditions under which it was tested (working on my machine, ship it!), but that will cause seemingly random problems once it hits users.

No, I still think your missing some of it. I am not advocating that what I said is the solution for everything.

Someone said that slicing UTF-8 strings leads to string corruption and endorsed the Python 3 frankenstien unicode type as a way to avoid it. I just gave a way of preventing that.

Now you argued that a novice programmer would fail to implement it properly. So you're comparing my method implemented by a novice programmer to a method implemented by profesional compiler writers. That hardly seems fair. :)

So my argument is that if my method were to be implemented by professional compiler writers it would prevent corrupted strings while still using UTF-8 as the internal representation.

> I basically undid all his changes, and wrapped all COBOL string splicing to call a function that always split a string at a valid position - truncating invalid bytes at the start/end as necessary.

> luckily for them they'd accidentally hired someone who had done lots of multilingual programming before.

So an expert programmer implemented a string splitting function that didn't corrupt strings. :D

> but even you made a mistake in your first example of what to do

I writing this on an iPad while watching TV and playing a game on another android tablet while looking at the wikipedia UTF-8 article on a tiny phone screen while a little white dog is trying to bite my fingers (wish I was making this up). Not exactly my usual programming environment. ;)

sigh if only it was novice programmers making these mistakes :-/

The stuff I've seen in some people's multithreaded code just makes me want to cry.

All you need to do is index/slice a string half-way through any character that is outside Unicode's Basic Multilingual Plane

I get that this might seem pedantic, but it's important to be pedantic about this, otherwise misconceptions and ambiguities occur e.g. 'just use wide characters' - the definition of which changes depending on the platform.

Second of all, "wide enough" for all intents and purposes means 32 bits. Technically Unicode only needs 21 bits to cover the currently defined codespace, but computers don't deal well with that and so 32bits is the minimum "wide enough" character size.

This creates a lot of wasted space and memory, not to mention pushes medium length strings across cache line boundaries for very little benefit - the ability to directly index/slice strings without accidentally corrupting data.

Now obviously you want to avoid accidentally corrupting data, the tradeoff comes down to whether you need direct, arbitrary indexing, or if it's worth doing some processing to determine the correct place to split in order to make space gains.

The technical world has come down overwhelmingly in favour of the latter, and that's why you see hardly anyone using utf-32. It's simply not as good a solution for most real world concerns.

A "string" means "a sequence of characters". Wide characters (or the equivalent interface) preserve this property.

Graphemes operate at a higher level than characters. You could construct a grapheme-strings, I suppose, but that has tons of edge cases, and if you don't like character-strings, I doubt you will like grapheme-strings.

The term character has many meanings. Graphemes are characters and that's what most users expect, something that's displayed as a single graphical unit.

"As with glyphs, there is no one-to-one relationship between characters and code points. What an end-user thinks of as a single character (grapheme) may in fact be represented by multiple code points; conversely, a single code point may correspond to multiple characters."

If we chop UTF-8, we can end up with bad characters, or possibly invalid overlong forms.

If you chop in the middel of a code unit then you end up with U-FFFDs. In both cases the visual representation has been altered.

As I wrote elsewhere it is easy for the slice routines of a language to check to see if the programmer tried to slice in the middle of a code unit and either return an error or just advance to the start of a code unit.

Slicing a code-point-character string destroys only graphemes.

Clear win.

A code point string has other niceties, like being indexed by simple integers. If end is the index of the last code point of a grapheme, then the next grapheme starts at end + 1.

If end is the index of the last UTF-8 encoding of a code point, then the next grapheme does not start at end + 1.

We can have it so that it does by making end point to the last byte of the UTF-8 encoding of the code point; but then it doesn't point at the start of the character, recovering which is awkward.

The code uglification can be addressed by piling on abstractions: integer-like iteration gizmos that can be incremented and decremented thanks to function or operator overloading.

I feel that that level of abstraction has no place in character-level data processing, if anywhere, whose basic operations should be expressible tersely in a few machine instructions.

Also, we mustn't lose sight of what the T means in UTF-8: transfer. It's not called UPF-8 (the Unicode processing format in 8 bits).

Working with UTF-8 instead of with the objects that UTF-8 denotes is like working with a textual representation of Lisp s-expressions that still contain the parentheses and whitespace delimitation, and quotes around strings and so on, refusing to parse them to obtain the object which they represent. People who do this should immediately turn in their CS degrees.

All those other issues you refer to are addressed by more parsing. If you want the glyphs, the correct thing is to parse the code-point string and make a list or vector of glyph representations.

With that representation you can still break the text "carpet" into "car" "pet" which destroys semantics; that is dealt with by parsing into words.

Chopping lists of words destroys phrases; so parse phrases, and transform at the phrase level.

And so on.

I'd call it a slight improvement. And after the major step back of using 2 to 4 times more memory for strings I'd call it a net loss.

> A code point string has other niceties, like being indexed by simple integers.

Again, no benefit of this. The only argument I've heard here is to prevent bad slicing and I've shown a way to prevent that.

> Also, we mustn't lose sight of what the T means in UTF-8: transfer. It's not called UPF-8 (the Unicode processing format in 8 bits).

By this argument we can't use UTF-16 or UTF-32 for internal processing of strings either. Back to code pages then.

Many unicode languages work with UTF-8 or UTF-16 internally. So working with the "transfer format" is common practice.

While it may not be necessary to know who the languages your program in work under the hood, expert programmers do want/need to know. That way they can write better code, or switch to another language or get the language devs to improve their internal handling.

The programmer shouldn't have to know that a newline character is written as \n in a JSON string.

The JSON string "a\nb" take 6 characters to write, but it's length should be given as 3.

99% people want to manipulate a JSON model, not the JSON (or BSON) serialization itself. The 1% can still use a byte array and do whatever hacks they like.

A better example is if you want to find a newline in a string. If you do a find it in a UTF-16 string it may be position 8 and a find in UTF-8 may be position 12. Does it matter what the actual number is? NO. You just pass it to the next function or whatever.

Edit: see pjscott's comment below - it's by code point, not byte, but still not by character.

    $ python3
    Python 3.4.3 (default, Oct 14 2015, 20:28:29) 
    >>> s = "オンライン"
    >>> s
    'オンライン'
    >>> len(s)
    5
    >>> s [0]
    'オ'
    >>> s [1]
    'ン'
    >>> s [2]
    'ラ'
    >>> s[3]
    'イ'
    >>> s[4]
    'ン'

    $ python3
    >>> "위키백과"[1]
    '키'
    >>> "위키백과"[1] # Should be identical, right?
    'ᅱ'

That is perhaps the most succinct and accurate way I've heard to explain and justify why you're sounding like a wet blanket to people that may not understand, while acknowledging that you know how you sound, but there is a reason for it. I expect to use this in the future.

c = "é"

c[0], c[1]

It's the same phenomenon with Latin characters. (Extra bonus: for me, the combining acute accent character then combines in the terminal with the apostrophe that Python uses to delimit the string!)

Another idea to see the effect is "a" + "é"[1]. (The result is 'á'... and as in your examples, a precomposed "̈́é" is also available which doesn't exhibit any of these phenomena.)

    "difficult"[2]?