News for hackers. Stuff that matters.
Re:Base65536 – Unicode's answer to Base64 by colanderman on Saturday December 05, 2015 @10:13PM (#10684242)
This only makes sense if your metric is number of code points. You don't actually get better density with this.

Base64, encoded as either ASCII or UTF-8, stores 6 bits per byte; making it 75% efficient.

Base65536 encoded as UTF-8 (by far the most common) will split its characters about evenly between 3-byte and 4-byte encodings (it uses code points up to 165376 roughly evenly distributed [1]). That is 16 bits per 3.5 bytes, or 57% efficient. Much worse.

UTF-16 (what Javascript uses) is slightly better; you get a distribution about even between 2 and 4 bytes. That is 16 bits per 3 bytes, or 67% efficient. Better, but not as good as Base64 in ASCII or UTF-8. (However, it is better than Base64 in UTF-16, which is only 37.5% efficient.)

Stick with Base64 unless you're strictly counting code points (like tweets do).

[1] https://github.com/ferno/base65536/blob/master/get-block-sta...

Re:Base65536 – Unicode's answer to Base64 by arkadiyt on Saturday December 05, 2015 @10:31PM (#10684292)
"This only makes sense if your metric is number of code points."

That is precisely the author's metric. In the "why?" section they write: "I wanted people to be able to share HATETRIS replays via Twitter. Twitter supports tweets of up to 140 characters. "

Re:Base65536 – Unicode's answer to Base64 by colanderman on Sunday December 06, 2015 @07:45PM (#10687454)
Yes, I read the article and mentioned its use case of tweets in the last sentence of my post.
Re:Base65536 – Unicode's answer to Base64 by deathanatos on Sunday December 06, 2015 @02:24AM (#10684660)
> This only makes sense if your metric is number of code points.

Not quite! :) I wrote a similar tool I called baseunicode[1]. (Glad I'm not the only one who's discovered this!) The author here is using it for tweets (which oddly don't count bytes, they count code points); I wrote my version for copy/pasting data between terminals[2]: copying more than a screen's worth is a hassle, so "screen area" is the metric I'm trying to optimize, which ends up being subtly different from the article's, which is code points for a tweet. (for screen area in a terminal, some characters — CJK in particular — end up taking multiple columns; and combining code points (like acute accent) are basically free!)

In my case, it ended up being easier to use base64, because it's installed by default.

[2]: SCP is just a pain sometimes. If you need to read a file only privileged by root, AFAIK, it's out of luck. I'm usually going server-to-server, so I also need to remember to -3, which I inevitably forget. You can get around the root issue w/ `ssh source 'sudo cat /foo/bar' | ssh destination 'cat >foo', (or 'sudo bash -c "cat >foo" if you need root on the write…) but more work; often I find I'm already there in both terminals and I just want these bytes from this split in this other split, so I started doing `tar -cz <stuff> | base64`, copy, `base64 -d | tar -xz`, paste. And then screen size mattered, thus baseunicode.

[1]: https://github.com/thanatos/baseunicode

Re:Base65536 – Unicode's answer to Base64 by adimitrov on Sunday December 06, 2015 @12:12PM (#10685767)
> which oddly don't count bytes, they count code points

Not so odd, but very sensible! I'm always saddened by this oversight in SMS messages — they're 160 7bit characters, so 140 octets. But those 7bit characters are the GSM charset. If you want to write in other languages than Western European ones, your message will be encoded using UTF-16 — now you only have 70 characters. It might be OK for Chinese, but it's absolutely unusable for Cyrillic.

Of course, the limiting factor in SMS is architectural, whereas in Twitter, it's philosophical, so there's your difference.

Re:Base65536 – Unicode's answer to Base64 by Dylan16807 on Sunday December 06, 2015 @01:37PM (#10686052)
There's only so many ways you can slice a kilobyte. Measuring bits is just the way to do it for data transmissions.

Support for compact representation of more languages would have been nice. Something like an 8 bit mode with 2 byte escapes for setting code page and 3 byte escapes for arbitrary unicode characters. As a bonus you'd actually be able to fit more emoji in that mode, 46 vs. 35.

Re:Base65536 – Unicode's answer to Base64 by amptorn on Sunday December 06, 2015 @05:09AM (#10684850)
If screen area is your metric, and combining code points are free, I bet you could use [1] to add an arbitrary amount of data to a single character.

[1] http://www.eeemo.net/

Re:Base65536 – Unicode's answer to Base64 by colanderman on Sunday December 06, 2015 @07:46PM (#10687457)
I do this too. xclip is your friend.
Re:Base65536 – Unicode's answer to Base64 by asdfaoeu on Saturday December 05, 2015 @10:31PM (#10684289)
You could however used the code points in a weighted matter and get better efficiency.
Re:Base65536 – Unicode's answer to Base64 by therein on Sunday December 06, 2015 @01:06AM (#10684538)
Only if you are forced to define your "code points" as UTF-8. Maybe due to some dependencies. But it is usually the opposite, dependencies tend to limit you to ASCII if they ever.