News for hackers. Stuff that matters.
Re:Base65536 – Unicode's answer to Base64 by colanderman on Saturday December 05, 2015 @10:13PM (#10684242)
This only makes sense if your metric is number of code points. You don't actually get better density with this.

Base64, encoded as either ASCII or UTF-8, stores 6 bits per byte; making it 75% efficient.

Base65536 encoded as UTF-8 (by far the most common) will split its characters about evenly between 3-byte and 4-byte encodings (it uses code points up to 165376 roughly evenly distributed [1]). That is 16 bits per 3.5 bytes, or 57% efficient. Much worse.

UTF-16 (what Javascript uses) is slightly better; you get a distribution about even between 2 and 4 bytes. That is 16 bits per 3 bytes, or 67% efficient. Better, but not as good as Base64 in ASCII or UTF-8. (However, it is better than Base64 in UTF-16, which is only 37.5% efficient.)

Stick with Base64 unless you're strictly counting code points (like tweets do).

[1] https://github.com/ferno/base65536/blob/master/get-block-sta...

Re:Base65536 – Unicode's answer to Base64 by arkadiyt on Saturday December 05, 2015 @10:31PM (#10684292)
"This only makes sense if your metric is number of code points."

That is precisely the author's metric. In the "why?" section they write: "I wanted people to be able to share HATETRIS replays via Twitter. Twitter supports tweets of up to 140 characters. "

Re:Base65536 – Unicode's answer to Base64 by colanderman on Sunday December 06, 2015 @07:45PM (#10687454)
Yes, I read the article and mentioned its use case of tweets in the last sentence of my post.
Re:Base65536 – Unicode's answer to Base64 by deathanatos on Sunday December 06, 2015 @02:24AM (#10684660)
> This only makes sense if your metric is number of code points.

Not quite! :) I wrote a similar tool I called baseunicode[1]. (Glad I'm not the only one who's discovered this!) The author here is using it for tweets (which oddly don't count bytes, they count code points); I wrote my version for copy/pasting data between terminals[2]: copying more than a screen's worth is a hassle, so "screen area" is the metric I'm trying to optimize, which ends up being subtly different from the article's, which is code points for a tweet. (for screen area in a terminal, some characters — CJK in particular — end up taking multiple columns; and combining code points (like acute accent) are basically free!)

In my case, it ended up being easier to use base64, because it's installed by default.

[2]: SCP is just a pain sometimes. If you need to read a file only privileged by root, AFAIK, it's out of luck. I'm usually going server-to-server, so I also need to remember to -3, which I inevitably forget. You can get around the root issue w/ `ssh source 'sudo cat /foo/bar' | ssh destination 'cat >foo', (or 'sudo bash -c "cat >foo" if you need root on the write…) but more work; often I find I'm already there in both terminals and I just want these bytes from this split in this other split, so I started doing `tar -cz <stuff> | base64`, copy, `base64 -d | tar -xz`, paste. And then screen size mattered, thus baseunicode.

[1]: https://github.com/thanatos/baseunicode

Re:Base65536 – Unicode's answer to Base64 by adimitrov on Sunday December 06, 2015 @12:12PM (#10685767)
> which oddly don't count bytes, they count code points

Not so odd, but very sensible! I'm always saddened by this oversight in SMS messages — they're 160 7bit characters, so 140 octets. But those 7bit characters are the GSM charset. If you want to write in other languages than Western European ones, your message will be encoded using UTF-16 — now you only have 70 characters. It might be OK for Chinese, but it's absolutely unusable for Cyrillic.

Of course, the limiting factor in SMS is architectural, whereas in Twitter, it's philosophical, so there's your difference.

Re:Base65536 – Unicode's answer to Base64 by Dylan16807 on Sunday December 06, 2015 @01:37PM (#10686052)
There's only so many ways you can slice a kilobyte. Measuring bits is just the way to do it for data transmissions.

Support for compact representation of more languages would have been nice. Something like an 8 bit mode with 2 byte escapes for setting code page and 3 byte escapes for arbitrary unicode characters. As a bonus you'd actually be able to fit more emoji in that mode, 46 vs. 35.

Re:Base65536 – Unicode's answer to Base64 by amptorn on Sunday December 06, 2015 @05:09AM (#10684850)
If screen area is your metric, and combining code points are free, I bet you could use [1] to add an arbitrary amount of data to a single character.

[1] http://www.eeemo.net/

Re:Base65536 – Unicode's answer to Base64 by colanderman on Sunday December 06, 2015 @07:46PM (#10687457)
I do this too. xclip is your friend.
Re:Base65536 – Unicode's answer to Base64 by asdfaoeu on Saturday December 05, 2015 @10:31PM (#10684289)
You could however used the code points in a weighted matter and get better efficiency.
Re:Base65536 – Unicode's answer to Base64 by therein on Sunday December 06, 2015 @01:06AM (#10684538)
Only if you are forced to define your "code points" as UTF-8. Maybe due to some dependencies. But it is usually the opposite, dependencies tend to limit you to ASCII if they ever.
Re:Base65536 – Unicode's answer to Base64 by spiznnx on Saturday December 05, 2015 @08:26PM (#10684009)
I was going to complain about lack of benchmarking, but then I saw the "Why" section - I wonder what else could be shared using a single tweet.
Re:Base65536 – Unicode's answer to Base64 by userbinator on Saturday December 05, 2015 @09:20PM (#10684109)
256-byte demos come to mind:

http://www.pouet.net/prodlist.php?type%5B%5D=256b

A little effect in 256 bytes of binary code. Some of them are amazing:

http://www.pouet.net/prod.php?which=66380

Re:Base65536 – Unicode's answer to Base64 by TazeTSchnitzel on Saturday December 05, 2015 @08:30PM (#10684019)
base65536 can fit at most 280 bytes in a tweet.

Most tweets I make are pure ASCII, which is 7 bit.

(280 * 8) / 7 = 320 septets

Well, 160 septets is the non-Unicode maximum length of an SMS.

So a tweet can fit two SMS messages in it!

Re:Base65536 – Unicode's answer to Base64 by mikeash on Saturday December 05, 2015 @08:41PM (#10684034)
You could do slightly better by not restricting yourself to the BMP. Full Unicode gives you almost 21 bits per code point, which is what Twitter counts. Round it down to 20 bits per "character" and you have 2800 bits = 350 bytes in a tweet, or 400 ASCII characters.
Re:Base65536 – Unicode's answer to Base64 by amptorn on Sunday December 06, 2015 @04:57AM (#10684831)
This is dangerous through, because sending a Unicode string through Twitter subjects it to NFC normalization which can alter its structure. If you want guarantees that this won't happen, your choices of code point are far more constrained, and the unassigned code points, which is most of them, can't even be used. I think this is the purpose of base65536gen.

https://www.npmjs.com/package/base65536gen

Re:Base65536 – Unicode's answer to Base64 by desdiv on Saturday December 05, 2015 @08:54PM (#10684049)
Unfortunately that won't work. Twitter doesn't support the full Unicode; they strip out any Unicode code points that's not used by "normal" people.
Re:Base65536 – Unicode's answer to Base64 by simplicio on Saturday December 05, 2015 @09:31PM (#10684132)
Are you sure? I just posted a tweet in Cuneiform, which is outside the BMP. So their view of "normal people" seems to encompass ancient Sumerians, at least.
Re:Base65536 – Unicode's answer to Base64 by mikeash on Saturday December 05, 2015 @09:02PM (#10684072)
Interesting, is there a list of what they strip out? I know they apply normalization so that would eliminate a few, but I think that wouldn't do too much on its own.
Re:Base65536 – Unicode's answer to Base64 by TazeTSchnitzel on Saturday December 05, 2015 @09:00PM (#10684063)
> they strip out any Unicode code points that's not used by "normal" people.

Such as?

Re:Base65536 – Unicode's answer to Base64 by goodplay on Saturday December 05, 2015 @09:36PM (#10684143)
LTR and RTL marks (all of them), which are used when the default Unicode bidi algorithm fails to display two intertwined scripts correctly. They are especially useful when writing short technical tweets.

I bet normal people would use them too if they knew about them and were given a chance, but normal people can't because twitter doesn't allow codepoints that aren't used by normal people.

Re:Base65536 – Unicode's answer to Base64 by TazeTSchnitzel on Saturday December 05, 2015 @09:48PM (#10684165)
Stripping out LTR and RTL override might be reasonable because they're frequently abused to screw up website layouts. But stripping all of the marks is just discrimination against people who need to write bi-directional text.
Re:Base65536 – Unicode's answer to Base64 by cbd1984 on Saturday December 05, 2015 @11:25PM (#10684402)
> Stripping out LTR and RTL override might be reasonable because they're frequently abused to screw up website layouts.

As long as Twitter allows Zalgo it isn't that concerned about website layouts.

(Zalgo: Rampant abuse of stacking combining diacritical marks above and below the line. It can get really, really... uh... impressive, in a Lovecraftian typewriter art kind of way.)

Re:Base65536 – Unicode's answer to Base64 by goodplay on Sunday December 06, 2015 @02:52AM (#10684691)
The issue of effecting web layout can be resolved by simply appending the appropriate resetting codepoint to the tweets. (U+202C for bidi overrides)
Re:Base65536 – Unicode's answer to Base64 by amptorn on Sunday December 06, 2015 @05:07AM (#10684847)
Well, null, to start with. Other ASCII control characters. Consecutive line breaks, too.
Re:Base65536 – Unicode's answer to Base64 by TazeTSchnitzel on Saturday December 05, 2015 @08:25PM (#10684005)
HATETRIS is pure evil. Try it.
Re:Base65536 – Unicode's answer to Base64 by haddr on Saturday December 05, 2015 @08:43PM (#10684038)
i got to 4 lines... incredible!
Re:Base65536 – Unicode's answer to Base64 by Dylan16807 on Saturday December 05, 2015 @10:25PM (#10684274)
Base65536 that isn't actually 65536? Odd.

Base32k is a similar idea, but it stores exactly 15 bits per codepoint. It uses three ranges of unicode, going for simplicity.

Edit: Ah, I see, it uses 65536 + 256 codepoints across two planes for different numbers of bytes. Which means it is simple in its own way, but it's also wasting a significant amount of its code space.

Re:Base65536 – Unicode's answer to Base64 by amptorn on Sunday December 06, 2015 @04:59AM (#10684833)
> Base65536 that isn't actually 65536? Odd.

Well, Base64 isn't actually 64 either.

Re:Base65536 – Unicode's answer to Base64 by vorg on Saturday December 05, 2015 @11:51PM (#10684443)
Why not use the Chinese characters (Kanji) to encode the moves in Chinese/Japanese semantics, instead of Base65536 ? It might not be as compact, but it would still be far more compact than Ascii AND you'd be able to read it without decoding software if you learn how to read a little Chinese or Japanese.
Re:Base65536 – Unicode's answer to Base64 by comex on Sunday December 06, 2015 @03:03AM (#10684701)
Because you couldn't find a meaningful way to assign Chinese characters to represent each possible combination of two of {left, right, down, rotate}, which is what the previous hex encoding mentioned can do, let alone do better. (As a novice, I can think of relatively reasonable ways to represent some of them, but no way to account for order. For example, "down left" could be "southwest" via the Zodiac animal, "left right" could be "horizontal", and "down rotate" could be "screw"... but that wouldn't help with "left down", "right left", etc.)
Re:Base65536 – Unicode's answer to Base64 by thaumasiotes on Sunday December 06, 2015 @02:30AM (#10684671)
It is significantly easier to write a format decoder than to learn to read, much less write, Chinese or Japanese.

Also, on a pedantic note, kanji are Japanese characters. Chinese characters are hanzi.

For most characters, they're the same, but there are a few differences.

Re:Base65536 – Unicode's answer to Base64 by vorg on Sunday December 06, 2015 @06:12PM (#10687135)
Fair point.

As for the pedantic note, kanji is an English word whereas hanzi isn't, so that's why I used it.

Re:Base65536 – Unicode's answer to Base64 by thaumasiotes on Sunday December 06, 2015 @07:09PM (#10687340)
eh... I think most people who recognize the word "kanji" at all are pretty well aware that it's a foreign word, and are likely to think of it that way. I do have a dictionary with an entry for "kanji", but that dictionary includes plenty of other foreign words too, like "haole" [Hawaiian term for non-Hawaiians, particularly (often pejorative) white people] (it doesn't have "hanzi", but it does have an entry for Han). I also have dictionaries that don't have an entry for "kanji".
Re:Base65536 – Unicode's answer to Base64 by javajosh on Saturday December 05, 2015 @09:09PM (#10684081)
Suggest: change name to Base64k.
Re:Base65536 – Unicode's answer to Base64 by jonesetc on Saturday December 05, 2015 @09:32PM (#10684134)
I really like this, but I'm not sure that the joke is obvious enough. Also, Base64Ki for more precision.
Re:Base65536 – Unicode's answer to Base64 by javajosh on Saturday December 05, 2015 @10:46PM (#10684337)
Well maybe it would be less jokey if you called it BaseUTF8 and write it not use the code points that require more than 8 bits. :)
Re:Base65536 – Unicode's answer to Base64 by DonHopkins on Saturday December 05, 2015 @09:32PM (#10684136)
Is ROT32768 Unicode's answer to ROT13?
Re:Base65536 – Unicode's answer to Base64 by ygra on Sunday December 06, 2015 @02:48AM (#10684686)
It would surely solve that pesky problem that double-ROT13 is so insecure. (Hint: Unicode is larger than the BMP.)
Re:Base65536 – Unicode's answer to Base64 by lyinsteve on Sunday December 06, 2015 @01:28AM (#10684572)
Why did I just implement this in Swift? https://github.com/harlanhaskins/Base65536.swift
Re:Base65536 – Unicode's answer to Base64 by 2bitencryption on Saturday December 05, 2015 @09:13PM (#10684092)
This reminds me of "usenet binaries."

The very existence of binaries on usenet still makes me chuckle. It's such a strange medium for binaries, and yet it was so prolific.

Re:Base65536 – Unicode's answer to Base64 by inopinatus on Saturday December 05, 2015 @09:42PM (#10684154)
NNTP is a general-purpose, globally meshed & distributed, channelized, eventually-consistent pub/sub bus. With optional ordering, GC, and replay semantics, and robust and stable implementations of both client and server protocol available in a variety of language bindings.

So the only real surprise is that it isn't everything's preferred loose-coupling substrate, but we do love reinventing the wheel.

Re:Base65536 – Unicode's answer to Base64 by userbinator on Saturday December 05, 2015 @09:26PM (#10684119)
It's not that strange if you think of things like email attachments... or do you also consider that to be "a strange medium for binaries"?
Re:Base65536 – Unicode's answer to Base64 by byuu on Saturday December 05, 2015 @10:32PM (#10684294)
Is there no end to the horrors spawned through Twitter's continued arbitrary limitation of 140 code points per tweet? :P
Re:Base65536 – Unicode's answer to Base64 by fooyc on Sunday December 06, 2015 @01:08PM (#10685953)
Reminds me of this: https://www.flickr.com/photos/quasimondo/3518306770/in/set-7...

This is an attempt at encoding images in a tweet, by using Unicode.

Re:Base65536 – Unicode's answer to Base64 by AndyKelley on Saturday December 05, 2015 @09:51PM (#10684177)
when would you ever use this? if you ever transmit unicode data, you have to encode it first. So one would probably encode this as UTF-8, which is binary. how about skip the middleman and transmit the binary directly?
Re:Base65536 – Unicode's answer to Base64 by colanderman on Saturday December 05, 2015 @10:30PM (#10684288)
This makes sense only if (a) your transport medium will only transfer graphical Unicode code points, and (b) you are interested in minimizing code point count as opposed to byte count.

If (a) is not true, you are better off just using binary as you state.

If (b) is not true, you are better off using Base64 if the underlying encoding is ASCII-compatible (e.g. ASCII, ISO-8859, or UTF-8), or Base4096 if the underlying encoding is UTF-16.

In the case of tweets, both (a) and (b) are true, making this somewhat worthwhile.

Re:Base65536 – Unicode's answer to Base64 by meepmorp on Saturday December 05, 2015 @10:00PM (#10684198)
It's explained in the linked post - people wanted to share HATETRIS replays through Twitter, and this helps that happen.
Re:Base65536 – Unicode's answer to Base64 by SixSigma on Sunday December 06, 2015 @03:45AM (#10684745)
Turn into image

Use pic.Twitter.com

Get 5Mb of data storage

Re:Base65536 – Unicode's answer to Base64 by jtwebman on Saturday December 05, 2015 @10:03PM (#10684206)
I'll stick to base64 or even better base85 (Ascii85).
Re:Base65536 – Unicode's answer to Base64 by Dylan16807 on Saturday December 05, 2015 @10:20PM (#10684261)
I've used base85 twice. Once internal to a database where it did stupid things with binary data, that went fine. Once for exporting data, that ended up breaking at random when formatters saw the tempting asterisks inside. I'd prefer base64 for most cases.