Database character sets and collations explained – why utf8 is not UTF-8

Re:Database character sets and collations explained – why ut... by tomwojcik on Thursday January 06, 2022 @03:52AM (#29820991)

I've been struggling with fixing some edge cases with emojis in latin1 encoded DB this past couple of days and what you mention here is the most important thing I found yesterday (conincidence).

Unicode strings are made up of Unicode code points, but some code points combine with others to make graphemes (single visual characters). Some emojis, like flags (U+1F1EA U+1F1E6 ), are made of two code points.

EDIT: Flag unicode.

[ Parent | on HN ]

Re:Database character sets and collations explained – why ut... by chrismorgan on Thursday January 06, 2022 @05:29AM (#29821794)

Correction: Unicode strings are made up of scalar values, not code points. The difference is that accursed mess that UTF-16 foist upon us, surrogates;oU+D800–U+DFFF are code points, but are not valid scalar values and cannot occur in Unicode strings (even if things like JavaScript are stupid in allowing potentially-malformed UTF-16, which is incompatible with Unicode, and Python similarly stupid in having strings that are not Unicode but are instead sequences of code points).

[ Parent | on HN ]

Re:Database character sets and collations explained – why ut... by tomwojcik on Thursday January 13, 2022 @04:25AM (#29918609)

Thanks. I think I will need to read a proper book / article on this subject as encoding in general keeps haunting me periodically.

[ Parent | on HN ]