import sys
import regex
print(regex.match("\X{,12}", sys.argv[1]).group())
with the regex[1] package that should be in the stdlib Any Day Now™.I wouldn't hold your breath. The issue tracker[1] suggests 3.7 or 3.8 as optimistic. Guido made some comment somewhere relatively recently, but I can't find where. It's entirely possible it will never actually happen; time doesn't seem to have made people more enthusiastic.
It's a shame, because the new module is awesome.
Ruby's unicode_utils gem has a nice implementation of the standard grapheme cluster segmentation algorithm, and Python's wrapper around ICU works quite well. Go's concept of runes is certainly an improvement, but it doesn't handle combining characters out of the box...
The good news is Unicode 8 will make them way more frequent! (alternate emoji skin colors are specified via combining characters) much as Unicode 6 made astral characters way more "in your face" (by standardising emoji in the SMP)
sub MAIN($s) { say $s.substr(0,12) }
$ perl6 test.p6 ǎěǐǒǔa̐e̐i̐o̐u̐ȃȇȋȏȗ
ǎěǐǒǔa̐e̐i̐o̐u̐ȃȇCould you retry with the input "देवनागरीदेवनागरी"?
sub MAIN($s) {
say "{$s.chars}: $s";
my $b = $s.substr(0,12);
say "{$b.chars}: $b";
}
$ perl6 hn-test2.p6 देवनागरीदेवनागरी
16: देवनागरीदेवनागरी
12: देवनागरीदेवनThe new string is composed of 10 user-visible characters (5 character repeated twice) but 16 codepoints (and this time I carefully checked that there was no precomposed version):
DEVANAGARI LETTER DA
DEVANAGARI VOWEL SIGN E
DEVANAGARI LETTER VA
DEVANAGARI LETTER NA
DEVANAGARI VOWEL SIGN AA
DEVANAGARI LETTER GA
DEVANAGARI LETTER RA
DEVANAGARI VOWEL SIGN II
DEVANAGARI LETTER DA
DEVANAGARI VOWEL SIGN E
DEVANAGARI LETTER VA
DEVANAGARI LETTER NA
DEVANAGARI VOWEL SIGN AA
DEVANAGARI LETTER GA
DEVANAGARI LETTER RA
DEVANAGARI VOWEL SIGN II
Operating on codepoints, both versions cut after the second DEVANAGARI LETTER NA (न) breaking that grapheme cluster (it should be ना) and not displaying the final two clusters ग and री.Yes and no. Yes, because the in-development Rakudo compiler is clearly currently giving the wrong result, and no because it operates on grapheme clusters (but has bugs).
(You can work with codepoints if you really want to but the normal string/character functions that use the normal string type, Str, work -- or more accurately are supposed to work -- on the assumption that "character" == grapheme cluster; afaik it's supposed to match the Unicode default Extended Grapheme Cluster specification.)
Fwiw I've filed a bug: https://rt.perl.org/Ticket/Display.html?id=125927
I don't think so. In my tests standard python (2.7 and 3.5) ignores grapheme clusters.
$ perl -CADS -E'say $ARGV[0] =~ /(\X{5})/' देवनागरीदेवनागरी
देवनागरी
Length of input string is: 10 graphemes, 16 codepoints, 48 octets (UTF-8).Length of output string is: 5 graphemes, 8 codepoints, 24 octets (UTF-8).
That's not true, Python 3 uses codepoint-based indexing but it will break if combining characters are involved. For instance:
because there is no precombined version of the multi-codepoint grapheme clusters so some of these 10 user-visible characters takes more than a single you end up with 8 user-visible characters rather than the expected 10.edit: the original version used the input string "ǎěǐǒǔa̐e̐i̐o̐u̐ȃȇȋȏȗ" where clusters turn out to have precomposed versions after all. Replaced it by devanāgarī repeated once (in the devanāgarī script)