News for hackers. Stuff that matters.
Re:Unicode is harder than you think by jameshart on Tuesday July 25, 2023 @09:04PM (#36871878)
Ask a human.

Failing that, ask an AI.

There's no way to take an arbitrary utf-8 string and shorten it to <= 4000 bytes without losing something. Heck, even ASCII has \r\n which can leave you in a bad place if you cut it off in the middle.

Even if you do the Unicode stuff right and make sure you don't break a grapheme cluster, you're still cutting words in half, which, if you're doing something like pulling AP headlines and truncating them to fit on a screen, can have embarrassing consequences for one like 'FDA Chief Exposes Butter Industry Corruption'. Even if you make sure to break on word boundaries you're still at risk of turning 'US Navy Fires Nuclear Weapons Program Chief' into 'US Navy Fires Nuclear Weapons' on your news ticker.

Strings are language. Language is hard.

There are no comments on this comment yet.