Apple adding those probably caused more upgrades to utf8mb4 than any amount of pleading from languages that actually needed them ever would have.
Sounds like you’re looking at how many scalar values are in an extended grapheme cluster (e.g. U+1F635 U+200D U+1F4AB dizzy face + zero width joiner + dizzy symbol, that’s one “character” made up of three scalar values).
The thing in question here is UTF-8 code units (bytes), and how many UTF-8 needs to encode a single scalar value. UTF-8 needs four bytes for anything above U+FFFF, and almost all emoji are above that.
Unicode strings are made up of Unicode code points, but some code points combine with others to make graphemes (single visual characters). Some emojis, like flags (U+1F1EA U+1F1E6 ), are made of two code points.
EDIT: Flag unicode.
So you can't use half the values of a byte when encoding code points outside of ASCII, and the remaining 128 values need to be split between starting bytes and continuation bytes. At best you're going to fit about 6 bits into each byte, and unicode spans the equivalent of just over 20 bits, so you need 4 bytes max.
utf-8 opts to keep encoding very simple, and in 2-4 bytes it can store 11, 16, and 21 bits respectively.
If you discarded ASCII compatibility, you could create a 1-3 byte self-synchronizing unicode encoding. But at that point maybe you should be compressing your text instead.
Or you could weaken the compatibility and create "utf-1 but better" to fit into 3 bytes.
Sorry, you can’t squeeze Unicode in that way, even unsynchronised: you fall just over 2½ bits short. No, reserving the 95 printable ASCII characters scuttles your hopes, meaning you get less than 1½ bits of that byte to use for the rest of Unicode, when you needed a touch over 4 bits.
To represent all the rest: Unicode scalar values minus printable ASCII is U+0000–U+10FFFF minus U+D800–U+DFFF minus U+0020–U+007E: 1,111,969 possibilities, log₂(1111969) ≈ 20.085 bits of entropy. (Removing the 2048 surrogates and 95 printable ASCII characters barely made a dent, under 0.003 bits, but I had to remove it for informational correctness.)
95 printable ASCII characters, log₂(95) ≈ 6.57 bits. Subtract that reservation from the three bytes available and you’ve got about 17.43 bits left for the rest of Unicode, but you needed over 20.
[Edit: I observe you’ve removed what I was quoting before I posted my comment. Still fun mathematics.]
> If you discarded ASCII compatibility, you could create a 1-3 byte self-synchronizing unicode encoding.
Not quite. If you discarded ASCII compatibility, then you’d have almost four bits to spare, which is enough for a self-synchronising fixed-width 3-byte encoding (reserving the first bit of each byte to signal if it’s a start byte) or for an unsynchronised variable-width encoding of some kind; but to make it even 2-or-3 or 1-or-3 (rather than 1–3) bytes wide variable-width, you’d need four bits (3 + log₂ 2) for self-synchronisation, and you’re about 0.085 bits short. (1–3 bytes variable-width would require 3 + log₂ 3 ≈ 4.58 bits.)
—⁂—
I just got nerd-sniped, didn’t I?
So it depends on exactly what we mean by compatible, and that's why I removed it.
But in particular, the encoding I had in mind when I wrote that was one that allows most ASCII characters as the middle byte of a triplet. That way whether you truncate from beginning or end you'll never have a rogue byte successfully decode.
So something like: Lead bytes from C0-FF, middle bytes from 00-BF, final bytes from 00-1F or 7F-BF.
If you also exclude null, tab, newline, and line feed, that gives you 64 x 188 x 93, which is just enough for every code point.
But arguably that's not compatible enough because you couldn't search for single ASCII characters with a dumb byte-wise algorithm.
> Not quite. If you discarded ASCII compatibility, then you’d have almost four bits to spare, which is enough for a self-synchronising fixed-width 3-byte encoding (reserving the first bit of each byte to signal if it’s a start byte) or for an unsynchronised variable-width encoding of some kind; but to make it even 2-or-3 or 1-or-3 (rather than 1–3) bytes wide variable-width, you’d need four bits (3 + log₂ 2) for self-synchronisation, and you’re about 0.085 bits short. (1–3 bytes variable-width would require 3 + log₂ 3 ≈ 4.58 bits.)
I don't know where you got that equation but it's not the right one for the situation.
Here's a really simple encoding just to disprove it: Leading bytes encode the plane (17 options) plus two more bits into decimal values 0 through 67. Continuation bytes encode 7 bits into decimal values 128 through 255. 17 planes + 2 bits + 7 bits + 7 bits, perfect fit. The remaining values, 68-127, can be used for single byte (and two byte) encodings.
But also the definition I was using for self-synchronizing doesn't seem to match the one on wikipedia. My intent was a code where you can guarantee sync after a specific small number of bytes (unlike UTF-1). In particular, if you let standalone bytes and continuation bytes overlap each other, you can still guarantee sync within 3 bytes. This gives you lots of space to play with, and you could make a very useful encoding. You could have >150 single byte codepoints and thousands of double byte codepoints and fit everything else into three bytes.
I wonder what HN's rules are with sanitizing Unicode, as they won't allow emojis?
HN removes things that are officially marked as emoji, normalises whitespace to space (which annoys me because I sometimes want to type things like narrow no-break space in equations which it makes wider and loses the no-breakyness of), and that’s all I’ve noticed. Control characters probably get some treatment or other too.
I think there is a lesson here - maybe I'll find it after my coffee
But one thing I haven't found out is why some emojis did go through. The basic ones like a simple :smile: it seemed to me. Could it be that some only need 3 bytes and when more and more emojis got released, they went into 4 bytes?
Edit: HN stripped the emojis from my comment... I would put a rolling eyes emoji here if I could.