News for hackers. Stuff that matters.
Re:Unicode is harder than you think by MadVikingGod on Tuesday July 25, 2023 @08:06PM (#36871424)
The number of bytes is still hard to do right. For example if I have a payload that can only be 4000 bytes, how do I take an arbitrary utf-8 string and get one that is <= 4000 bytes and doesn't cut off a graphmeme cluster?
Re:Unicode is harder than you think by jameshart on Tuesday July 25, 2023 @09:04PM (#36871878)
Ask a human.

Failing that, ask an AI.

There's no way to take an arbitrary utf-8 string and shorten it to <= 4000 bytes without losing something. Heck, even ASCII has \r\n which can leave you in a bad place if you cut it off in the middle.

Even if you do the Unicode stuff right and make sure you don't break a grapheme cluster, you're still cutting words in half, which, if you're doing something like pulling AP headlines and truncating them to fit on a screen, can have embarrassing consequences for one like 'FDA Chief Exposes Butter Industry Corruption'. Even if you make sure to break on word boundaries you're still at risk of turning 'US Navy Fires Nuclear Weapons Program Chief' into 'US Navy Fires Nuclear Weapons' on your news ticker.

Strings are language. Language is hard.

Re:Unicode is harder than you think by Dylan16807 on Tuesday July 25, 2023 @10:35PM (#36872469)
Re:Unicode is harder than you think by eviks on Wednesday July 26, 2023 @05:51AM (#36875155)
Count by grapheme clusters and stop right before it exceeds the limit?
Re:Unicode is harder than you think by zamadatix on Wednesday July 26, 2023 @10:11PM (#36888021)
Isn't that the length of the string? Also there are various things about Unicode that make doing this "right" for an input much harder than it sounds.
Re:Unicode is harder than you think by eviks on Thursday July 27, 2023 @12:19AM (#36888855)
string length is a poorly defined concept, but yes, counting real characters (grapheme clusters) counts

And what "various things" do you mean that are on top the char split?

Re:Unicode is harder than you think by dhosek on Thursday July 27, 2023 @11:28AM (#36895050)
The rules for grapheme clusters are well-defined and documented. I’ve only implemented a forward iterator on graphemes in the finl_unicode rust crate, but backwards would not be that difficult either. If I’m looking at a byte stream, I have to first make sure that I’m not in the middle of a multi-byte sequence (easily enough accomplished), you need to do some forwards and backwards iteration to determine where the grapheme boundaries are, and likely you will want a library to do this for you, but it’s something that should take a reasonably skilled programmer in about a day or two.