News for hackers. Stuff that matters.
Re:A Programmer’s Introduction to Unicode by masklinn on Saturday March 04, 2017 @11:56AM (#13791687)
> Is there any reasonably popular encoding that is "hole free / complete" in the sense any sequence of bytes has a valid decoding ?

That's a property of the ISO-8859 encodings, it turns out to be pretty terrible as you have no way of distinguishing utter garbage/nonsense from actual text.

> I use B85 or B64 when needed but was wondering if there is a Unicode encoding that will do the job.

Base64 and 85 are pretty much the opposite of Unicode encodings. Encodings turn text (abstract unicode codepoints) into actual bytes you can store, Base64 and Base85 turn bytes into ASCII or ASCII-compatible text.

Re:A Programmer’s Introduction to Unicode by srean on Saturday March 04, 2017 @12:11PM (#13791777)
> Base64 and 85 are pretty much the opposite

I know and that's the purpose I use it for: to smuggle in binary data as text, but no reason why an encoding scheme should not be dual purpose. Thanks for the tip about ISO-8859

Re:A Programmer’s Introduction to Unicode by edblarney on Saturday March 04, 2017 @04:06PM (#13792936)
In b64, you're going to want to pick a short set of characters that can be encoded clearly and simply, i.e. ascii chars. And the shorter the better, i.e. even in 6 bits.

You take the first six bits of your binary data and convert to some ascii char mapping. And then the the next 6 bits and so on.

You can't do that with unicode, and it wouldn't make sense for any other encoding standard.

They are completely different things.

Re:A Programmer’s Introduction to Unicode by Moru on Saturday March 04, 2017 @09:42PM (#13794294)
Mabe srean wants a more efficient way of base64 a binary file into some text-only media like email. This depends totally on where you want to put it. If it's an email you are pretty stuck with base64, if it's a string that nothing else touches, you can use the binary data directly (eg: iso-8859) :)
Re:A Programmer’s Introduction to Unicode by srean on Monday March 06, 2017 @12:59PM (#13804541)
That's indeed right. B85 is already a little more efficient than B64, but was wondering if one could abuse Unicode for this.

Its a really silly, stupid situation I need this for, exchanging data with Python and I can only use Unicode strings.

Re:A Programmer’s Introduction to Unicode by masklinn on Thursday March 09, 2017 @06:23AM (#13828568)
> That's indeed right. B85 is already a little more efficient than B64, but was wondering if one could abuse Unicode for this.

Sure, kinda: https://github.com/pfrazee/base-emoji (it's a base256 using emoji), but then you still need to encode that text, which is going to require 4 bytes per symbol, so I'm not sure you're going to get any actual gain over B64/B85 in the end. There's also the option of using a subset of the U+0100~U+07FF range (though it contains a diacticial block which may not be ideal) as it encodes to 2 bytes in both UTF-8 and UTF-16 (though there are diacritics in these blocks, and some of the codepoints are reserved but not allocated so…).