> Mostly, `io` should handle this in both 3/2. You might need to help it get the right encoding in 2.
Does io handle stdio? I was referring to standard input and standard error/output. How do you read/write Unicode in a cross-2/3 way from/to standard input/output/error without adding your own translation layer?
>> Unicode file paths
> This is going to be a mess in any language, because file paths really aren't text.
Sorry I need to be more clear. That general mess is not the aspect of it I was referring to. I'm specifically referring to a 2/3 compatibility mess.
I meant that, for example, to have any semblance of Unicode handling, in Python 2 you do os.listdir(u"."), whereas in Python 3 you do os.listdir(b"."). I know how to handle it in both Python 2 and Python 3 in a way that's Good Enough (TM), but how do I even get that with cross-compatible code? I'd need to write a translation layer of sorts for every single I/O function I might use.
>> Unicode sys.argv
> I think I'd say "do whatever 3 does".
Hmm okay thanks, I'll need to try to see what the implications are again. I think the problems I recalled from this may have been just been a result of the other issues, not sure.
I can't "forget about 2" though, it's still on Ubuntu LTS systems and there are still packages in 2 that haven't been ported to 3.
>> I feel like Python 3 completely wrecked strings instead of making them better.
> A clear separation of text and binary is needed in the long run, and makes other operations much clearer and saner. The pain you're feeling is introduced from the OS not having the same clarity.
Again, I think this is "cleaner" in theory, not in practice. What happened to the string_escape/unicode_escape nonsense I pointed out with the new system? Any rebuttal to that one? ;-)
In some cases of common OS-induced pain, I'd say "do whatever 3 does" in 2, since that'll make migration easier in the long run. (But I understand that can be hard, and I think my responses to your examples below even demonstrate that to be hard.)
To your specific pain points:
> Unicode stdio?
Mostly, `io` should handle this in both 3/2. You might need to help it get the right encoding in 2.
> Unicode file paths
This is going to be a mess in any language, because file paths really aren't text. On nix, they're byte strings that don't have nuls in them. Hopefully* they're encoded according to LANG, and hopefully LANG is a UTF-8 variant, but it isn't required, and it isn't required that two users on the same system use compatible LANGs, so you get a Tower of Babel. I really wish OSs would just start enforcing a. Unicode filenames, and b. no newlines in filenames; those two alone would make life so much easier.
Hopefully you've seen os.fsencode / fsdecode, but alas those aren't in 2, so I'm not sure they really help you. Often one is not really munging paths that much, and can just pass through whatever value/type you get, but it does happen, of course. (E.g., adding or removing extensions)
> Unicode sys.argv
This is also a pain point, since again, the underlying type in nix is a byte string without nuls. I'd hope it decodes w/ the LANG encoding, but since the user could easily tab-complete a filename, fsencode/decode might be more appropriate. I think I'd say "do whatever 3 does".
1 Jan 2020 is nearly here. Forget about 2 / assume UTF-8 in 2 and don't support anything else?
> I feel like Python 3 completely wrecked strings instead of making them better.*
A clear separation of text and binary is needed in the long run, and makes other operations much clearer and saner. The pain you're feeling is introduced from the OS not having the same clarity.