News for hackers. Stuff that matters.
Re:The History and rationale of the Python 3 Unicode model f... by mmastrac on Wednesday December 14, 2022 @11:09AM (#33985535)
Rust uses a solution similar to his original proposal: OsString/OsStr: https://doc.rust-lang.org/std/ffi/struct.OsString.html

I wonder if unicode needs a "undecodable byte escape" sequence for these raw sequences that cannot be mapped into the string space, allowing for a guaranteed round-trip between the OS and Unicode. For this to work, you'd need a guarantee that all codepage/unicode mappings are consistent and 1:1 which might not be possible.

Paths are tricky because they act as both an identifier and a handle, and the string encoding of the identifier and handle are _close but not exactly the same_ when unicode is in the mix. In the ASCII world, you can assume that this is the case.

Re:The History and rationale of the Python 3 Unicode model f... by LegionMammal978 on Wednesday December 14, 2022 @05:22PM (#33991506)
Python already has the "surrogateescape" error handler [0] that performs something similar to what you described: undecodable bytes are translated into unpaired U+DC80 to U+DCFF surrogates. Of course, this isn't standardized in any way, but I've found it useful myself for smuggling raw pathnames through Java.

[0] https://peps.python.org/pep-0383/

Re:The History and rationale of the Python 3 Unicode model f... by mmastrac on Wednesday December 14, 2022 @09:11PM (#33994167)
Thanks for this reference! It's exactly what I was thinking.
Re:The History and rationale of the Python 3 Unicode model f... by pvillano on Wednesday December 14, 2022 @07:54PM (#33993400)
wrt "undecidable byte escape" I think that wouldn't solve the problems with filenames. Files would have to enter Python land escaped which would mean my file really named \x00 would become \\x00

You can't escape only the broken filenames, because a valid unicode filename could contain an escape sequence, which would incorrectly be unescaped when returning to the OS

I think you really do need a non-str type that can be silently converted to str 99% of the time