> For example ISO 10646 ("Unicode") has multi-code point sequences denoting still a single character
I only know two programming languages that deal correctly with that: Swift and Perl 6. If you know more, please tell me.
Re:Grokking Python 3’s str
by steveklabnik on Saturday January 14, 2017 @02:05PM
(#13399952)
Rust separates this out; the standard library gives you bytes and Unicode Scalar Values. Grapheme cluster stuff is in a package on Cargo, maintained by the Servo team.
You can iterate over Grapheme clusters using the standard library: https://doc.rust-lang.org/1.3.0/std/str/struct.Graphemes.htm...
Re:Grokking Python 3’s str
by steveklabnik on Saturday January 14, 2017 @06:28PM
(#13401260)
Those are unstable docs for 1.3; they don't exist in today's Rust.
If you rely on libraries, there are ICU libraries for C and Java too, see for example http://userguide.icu-project.org/boundaryanalysis
Sounds like Rust follows in the tradition of C. Like you now always hear about hos C has no string type, just a bunch of char, will we hear in the future that, alas, Rust has no native support for grapheme clustering?
Re:Grokking Python 3’s str
by steveklabnik on Saturday January 14, 2017 @07:30PM
(#13401530)
The issue is, there's multiple ways to do the clustering, in my understanding. We picked the most core thing to be our most core thing. Text is hard.
Oh I get it didn't want to come across as flippant. Indeed text is hard.
I was sloppy in using the word "character" - strings are really sequences of codepoints. I've edited my post to reflect this. Of course, multiple codepoints may be needed to construct a single character, and that's by design.
Re:Grokking Python 3’s str
by tannhaeuser on Saturday January 14, 2017 @12:25PM
(#13399418)
But has Python 3 Unicode-aware string-length rather than byte-length functions (+ substring, case-folding, character classification functions etc.) for dealing with this? If it hasn't then nothing really is achieved for the general case by switching to 4-byte or any other fixed-byte unit, including the case of dealing with HTML 5 text.
See also perlgeek's question.
Yes, Python 3 does have Unicode-aware functions, as demonstrated in the post (via slices).
Go and Rust also have very good Unicode support, in addition to Swift and (apparently) Perl 6 as mentioned above.
Moreover, there are languages/scripts challenging the notion that a particular byte-unit corresponds to a single character.
Treating strings as byte sequences (with optional UTF-8 interpretation/checking, uppercase/lowercase conversion if the concept even applies, trim functions based on all Unicode spaces not just those characters in US-ASCII etc.) is entirely a defensible choice for a programming language.