It depends what your string is a string of.

Slicing by byte-offset is pretty unhelpful, given how many Unicode characters occupy more than one byte. In an encoding like UTF-16, that's "all of them" but even in UTF-8 it's still "most of them".

Slicing by UTF-16 code-unit is still pretty unhelpful, since a lot of Unicode characters (such as emoji) do not fit in 16 bits, and are encoded as "surrogate pairs". If you happen to slice a surrogate pair in half, you've made a mess.

Slicing by code-points (the numbers allocated by the Unicode consortium) is better, but not great. A shape like the "é" in "café" could be written as U+0065 LATIN SMALL LETTER E followed by U+0301 COMBINING ACUTE ACCENT. Those are separate code-points, but if you slice between them you'll wind up with "cafe" and an isolated acute accent that will stick to whatever it's next to, like this:́

When combining characters stick to a base character, the result is called a "grapheme cluster". Slicing by grapheme clusters is the best option, but it's expensive since you need a bunch of data from the Unicode database to find the edges of each cluster - it depends on the properties assigned to each character.

Doesn't splitting by grapheme cluster also depends on which version of the unicode standard you use, since new standards come with new combinations?

The standard answer is "don't". Just treat text is a blob, but the other question is what are you trying to accomplish?

- Are you trying to control the rendered length? In that case the perfect solution is actually rendering the string.

- Are you limiting storage size? Then you need to find a good split point that is <N bytes. This is probably done using extended grapheme clusters. (Although this also isn't perfect)

I'm sure there are other use cases as well. But at the end of the day try to avoid splitting text if it can be helped.

it think the only resonable rule for substring(m, n) is "don't"

So string is no longer a "string of characters", it is in fact a program (not Turing complete) that you need to execute.

Though substring(m, n) still makes sense in at least interactive text manipulation: how do you do copy/paste?

No it is not a program - at least not anymore than an ASCII string is a program.

It is just that there isn't a simple 1:1 correspondence between bytes and characters and glyphs as in unicode, so you cant just extract an arbitrary byte-sequence from a string and expect it to render correctly.

> there isn't a simple 1:1 correspondence between bytes and characters and glyphs

There isn't a simple 1:1 correspondence between anything at all. The only definitive thing about Unicode strings is the beginning where you should start your parsing.

Then the way things are supposed to be displayed to be Unicode-compliant look more like some virtual machine analyzing the code. How is this different from any other declarative language?

The safest path is to consider it a blob. There is some library that can render it magically and that's the only wise thing you can do. The internal structure is hard to understand. Also, definitions change over time. So, you better leave it all to professionals.

The thing about Unicode is.... anybody who tried to do it “more simple” would eventually just develop a crappier version of Unicode.

Unicode is complex because the sum of all human language is complex. Short of a ground up rewrite of the worlds languages, you cannot boil away most of that complexity... it has to go somewhere.

And even if you did manage to “rewrite” the worlds languages to be simple and remove accidental complexity I assert that over centuries it would devolve right back into a complex mess again. Why? Languages represent (and literally shape and constrain) how humans think and humans are a messy bunch of meat sacks living in a huge world rich in weird crazy things to feel and talk about.

There are definitely crappy things about Unicode that are separate from language.

- Several writing systems are widely scattered across multiple ‘Supplement’/‘Extended’/‘Extensions’ blocks.

- Operators (e.g. combining forms, joiners) are a mishmash of postfix, infix, and halffix. They should have been (a) in an easily tested reserved block (e.g. 0xF0nn for binary operators, 0xFmnn for unary), so that you could parse over a sequence even if it contains specific operators from a later version — i.e. separate syntax from semantics, and (b) uniformly prefix, so that read-ahead isn't required to find the end of a sequence (and dead keys become just like normal characters).

Not really. A Unicode string is more like a sequence of data built from simple binary structs, which belong to a smallish group of valid structs. Additionally, some but not all, of these structs can be used to infer the validity of subsequent structs in the sequence if your parsing in a more byte-at-a-time fashion. Alternately if your happy dealing with a little less forward compatibility and go for explicit enumeration of all groups of valid bytes you can be a lot more sure of things but it’s harder to make this method as performant as the byte-at-a-time method, which given the complete ubiquity of string processing in software... leads to the dominance of the byte-at-a-time method.

"So string is no longer a "string of characters""

It hasn't been for 30 years.

It is up to the implementation.

This is a good read on aspects of it:

https://hsivonen.fi/string-length/

Maybe have m,n refer to grapheme clusters instead of bytes/code points?

Apparently it's what Swift does when you try to get the length of a string. Though there's no more plain substring() since Swift 5, it was removed to indicate it's no longer O(1). You will get different results across languages though.