Aha ! thanks for the anecdote. My situation is indeed very similar, and yeah Python is involved, but that's not where I would put the blame.
Re:A Programmer’s Introduction to Unicode
by doubleunplussed on Friday March 10, 2017 @10:25AM
(#13839245)
Oh, I'm not saying Python is to blame in any sense, rather, that the core devs had the same problem as you and hit on a neat solution.
Oh ! sorry never thought you were saying Python was the source of the problem.
Since Python happened to be a common feature common between our two scenarios, wanted to caution other HN'ers not to put blame on Python. In my case Python was indeed doing the right thing.
https://www.python.org/dev/peps/pep-0383/
So with most systems being UTF8, this means that the encoding is "UTF8, except with bytes that would be invalid UTF8 mapping to this range of surrogates". So you've got an encoding with no holes, which is compatible with all valid UTF8 (since those surrogates are invalid UTF8).
It's not common, so it doesn't match that criterion of yours,but I think it's a great solution and hope it catches on. We need a solution for "text, except with graceful fallback when someone decided to put arbitrary bytes in there"