> Is there any reasonably popular encoding that is "hole free / complete" in the sense any sequence of bytes has a valid decoding ?

That's a property of the ISO-8859 encodings, it turns out to be pretty terrible as you have no way of distinguishing utter garbage/nonsense from actual text.

> I use B85 or B64 when needed but was wondering if there is a Unicode encoding that will do the job.

Base64 and 85 are pretty much the opposite of Unicode encodings. Encodings turn text (abstract unicode codepoints) into actual bytes you can store, Base64 and Base85 turn bytes into ASCII or ASCII-compatible text.

> Base64 and 85 are pretty much the opposite

I know and that's the purpose I use it for: to smuggle in binary data as text, but no reason why an encoding scheme should not be dual purpose. Thanks for the tip about ISO-8859

In b64, you're going to want to pick a short set of characters that can be encoded clearly and simply, i.e. ascii chars. And the shorter the better, i.e. even in 6 bits.

You take the first six bits of your binary data and convert to some ascii char mapping. And then the the next 6 bits and so on.

You can't do that with unicode, and it wouldn't make sense for any other encoding standard.

They are completely different things.

Mabe srean wants a more efficient way of base64 a binary file into some text-only media like email. This depends totally on where you want to put it. If it's an email you are pretty stuck with base64, if it's a string that nothing else touches, you can use the binary data directly (eg: iso-8859) :)

That's indeed right. B85 is already a little more efficient than B64, but was wondering if one could abuse Unicode for this.

Its a really silly, stupid situation I need this for, exchanging data with Python and I can only use Unicode strings.

> That's indeed right. B85 is already a little more efficient than B64, but was wondering if one could abuse Unicode for this.

Sure, kinda: https://github.com/pfrazee/base-emoji (it's a base256 using emoji), but then you still need to encode that text, which is going to require 4 bytes per symbol, so I'm not sure you're going to get any actual gain over B64/B85 in the end. There's also the option of using a subset of the U+0100~U+07FF range (though it contains a diacticial block which may not be ideal) as it encodes to 2 bytes in both UTF-8 and UTF-16 (though there are diacritics in these blocks, and some of the codepoints are reserved but not allocated so…).

Python hit this issue in the last few years after the transition to unicode-everything. Since the filesystem is supposed to have an encoding, but doesn't enforce it, Python needed to choose what to do with a filename that contained bytes that were invalid for the encoding. In order to make a round trip of decoding and encoding result in the same bytes, they decided to use certain surrogates to represent the invalid bytes:

https://www.python.org/dev/peps/pep-0383/

So with most systems being UTF8, this means that the encoding is "UTF8, except with bytes that would be invalid UTF8 mapping to this range of surrogates". So you've got an encoding with no holes, which is compatible with all valid UTF8 (since those surrogates are invalid UTF8).

It's not common, so it doesn't match that criterion of yours,but I think it's a great solution and hope it catches on. We need a solution for "text, except with graceful fallback when someone decided to put arbitrary bytes in there"

Aha ! thanks for the anecdote. My situation is indeed very similar, and yeah Python is involved, but that's not where I would put the blame.

Oh, I'm not saying Python is to blame in any sense, rather, that the core devs had the same problem as you and hit on a neat solution.

Oh ! sorry never thought you were saying Python was the source of the problem.

Since Python happened to be a common feature common between our two scenarios, wanted to caution other HN'ers not to put blame on Python. In my case Python was indeed doing the right thing.

Every 1-byte charset that uses high bits (i.e., not UTF-7, ASCII, or EBCDIC) has this property. Depending on how you feel about whether unpaired surrogates constitute a valid decoding, UTF-16 has that property as well.

It's not a very good property to have. UTF-8's invalid decodings means that it's very easy to detect if you're not using UTF-8: iconv's charset detector will rule certainty of UTF-8 if it sees just 3 valid UTF-8 multibyte sequences and no invalid sequences, and that's pretty much the only reliable detector.

Javascript. It uses bad UTF-16, where unpaired surrogates are allowed. They're meaningless, but allowed.