Re:Base65536 – Unicode's answer to Base64
by mikeash on Saturday December 05, 2015 @08:41PM
(#10684034)
You could do slightly better by not restricting yourself to the BMP. Full Unicode gives you almost 21 bits per code point, which is what Twitter counts. Round it down to 20 bits per "character" and you have 2800 bits = 350 bytes in a tweet, or 400 ASCII characters.
Re:Base65536 – Unicode's answer to Base64
by amptorn on Sunday December 06, 2015 @04:57AM
(#10684831)
This is dangerous through, because sending a Unicode string through Twitter subjects it to NFC normalization which can alter its structure. If you want guarantees that this won't happen, your choices of code point are far more constrained, and the unassigned code points, which is most of them, can't even be used. I think this is the purpose of base65536gen.
Re:Base65536 – Unicode's answer to Base64
by desdiv on Saturday December 05, 2015 @08:54PM
(#10684049)
Unfortunately that won't work. Twitter doesn't support the full Unicode; they strip out any Unicode code points that's not used by "normal" people.
Re:Base65536 – Unicode's answer to Base64
by simplicio on Saturday December 05, 2015 @09:31PM
(#10684132)
Are you sure? I just posted a tweet in Cuneiform, which is outside the BMP. So their view of "normal people" seems to encompass ancient Sumerians, at least.
Re:Base65536 – Unicode's answer to Base64
by mikeash on Saturday December 05, 2015 @09:02PM
(#10684072)
Interesting, is there a list of what they strip out? I know they apply normalization so that would eliminate a few, but I think that wouldn't do too much on its own.
Re:Base65536 – Unicode's answer to Base64
by TazeTSchnitzel on Saturday December 05, 2015 @09:00PM
(#10684063)
> they strip out any Unicode code points that's not used by "normal" people.
Such as?
Re:Base65536 – Unicode's answer to Base64
by goodplay on Saturday December 05, 2015 @09:36PM
(#10684143)
LTR and RTL marks (all of them), which are used when the default Unicode bidi algorithm fails to display two intertwined scripts correctly. They are especially useful when writing short technical tweets.
I bet normal people would use them too if they knew about them and were given a chance, but normal people can't because twitter doesn't allow codepoints that aren't used by normal people.
Re:Base65536 – Unicode's answer to Base64
by TazeTSchnitzel on Saturday December 05, 2015 @09:48PM
(#10684165)
Stripping out LTR and RTL override might be reasonable because they're frequently abused to screw up website layouts. But stripping all of the marks is just discrimination against people who need to write bi-directional text.
Re:Base65536 – Unicode's answer to Base64
by cbd1984 on Saturday December 05, 2015 @11:25PM
(#10684402)
> Stripping out LTR and RTL override might be reasonable because they're frequently abused to screw up website layouts.
As long as Twitter allows Zalgo it isn't that concerned about website layouts.
(Zalgo: Rampant abuse of stacking combining diacritical marks above and below the line. It can get really, really... uh... impressive, in a Lovecraftian typewriter art kind of way.)
Most tweets I make are pure ASCII, which is 7 bit.
(280 * 8) / 7 = 320 septets
Well, 160 septets is the non-Unicode maximum length of an SMS.
So a tweet can fit two SMS messages in it!