News for hackers. Stuff that matters.
Re:How UTF-8 Works by edflsafoiewq on Tuesday February 08, 2022 @01:35PM (#30262461)
Why is that useful?
Re:How UTF-8 Works by mananaysiempre on Tuesday February 08, 2022 @02:07PM (#30262946)
You can use FF as a sentinel byte internally (I think utf8proc actually does that?); given that FE never occurs, either, if you see the byte sequence corresponding to U+FEFF BYTE ORDER MARK in one of the other UTFs you can pretty much immediately tell it can’t possibly be UTF-8. (In general UTF-8, because of all the self-synchronization redundancy, has a very distinctive pattern that allows it to be detected with almost perfect reliability, and that is a frequent point of UTF-8 advocacy, which lends some irony to the fact that UTF-8 is the one encoding that Web browsers support but refuse to detect[1].) I don’t think there is any other advantage to excluding FF specifically, it’s not like we’re using punched paper tape.

[1] https://hsivonen.fi/utf-8-detection/

Re:How UTF-8 Works by xigoi on Tuesday February 08, 2022 @02:33PM (#30263351)
You can use 11000000 and 11000001 as sentinel bytes; since a sequence beginning with them can't possibly be minimal.