News for hackers. Stuff that matters.
Re:The complete guide to working with strings in modern Java... by crazygringo on Tuesday April 13, 2021 @12:38PM (#26795031)
Exactly, and emoji are outside the BMP, so it's not exactly an edge case, but the norm where two code units (UTF-16 double-bytes) are used to make one code point (Unicode character) [1].

And it gets even worse, when you consider that for many purposes you're not even interested in code points but in graphemes which can be sequences of code points -- e.g. a single visible emoji might actually be a sequence of 5 code points, represented by 8 UTF-16 code units, taking up 16 bytes [2]. Similarly a single accented character will often be two code points (letter plus combining diacritic).

If you want to split a string by graphemes -- e.g. to count its visible length, or delete its last visible character -- you can either use a library for it [3], or the relatively new Intl.Segmenter [4] which is in Chrome and Safari, but hasn't made it to Firefox [5].

Kind of amazing it's 2021 and you still can't calculate the number of visible characters (graphemes) in a string using native functions across all modern browsers.

[1] https://blog.jonnew.com/posts/poo-dot-length-equals-two

[2] https://www.contentful.com/blog/2016/12/06/unicode-javascrip...

[3] https://github.com/orling/grapheme-splitter

[4] https://github.com/tc39/proposal-intl-segmenter

[5] https://bugzilla.mozilla.org/show_bug.cgi?id=1423593

Re:The complete guide to working with strings in modern Java... by dwheeler on Tuesday April 13, 2021 @06:56PM (#26800537)
You're right about the graphemes. If you need it, you need it, but I recommend writing code that does not need to count graphemes if you can avoid them.

In many cases strings are best considered units that can be concatenated at will, but it's best if you avoid splitting them, and if you must split them, generally only split them on ASCII character boundaries. Don't consider "lengths" as something that has a meaning to humans (it doesn't), and don't assume that a "character" is a single JavaScript character (it isn't). If you normally just work strings as opaque sequences of "characters" that can be later displayed, you can avoid many complications (though obviously NOT all of them).

Re:The complete guide to working with strings in modern Java... by davethedevguy on Tuesday April 13, 2021 @01:26PM (#26795820)
Thank you for these comments. I didn't know about this at all!

I'll read up on it until I understand it, and then add something to the article that covers it.

Re:The complete guide to working with strings in modern Java... by crazygringo on Tuesday April 13, 2021 @03:18PM (#26797624)
This might also help you:

https://stackoverflow.com/questions/4547609/how-to-get-chara...

e.g. use Array.from() to at least process code points rather than code units, though that's still not graphemes.