News for hackers. Stuff that matters.
Re:UTF-8 Everywhere by samatman on Tuesday April 14, 2020 @05:39PM (#22871804)
Avoiding invalid UTF-8 is easy, almost trivial: just make sure you don't truncate in the middle of a code point.

The latter is fiendishly difficult to get right in all cases, the ugliest case being emoji flags. Being all-or-nothing on both sides of a ZWJ will get you most of the way there, however.

Re:UTF-8 Everywhere by smasher164 on Tuesday April 14, 2020 @05:54PM (#22871974)
It's not though. Replacing invalid byte sequences is not terribly difficult.

https://golang.org/src/strings/strings.go?s=15854:15900#L627.

Re:UTF-8 Everywhere by samatman on Tuesday April 14, 2020 @08:11PM (#22872995)
We are agreeing, the part I was indicating is difficult is 'displaying gibberish'.

Knowing what constitutes a grapheme cluster is detailed and frequently changes.