The issue w.r.t. paths is that paths have different types depending on the operating system. On Linux and most BSDs paths are bytes. On some Unices they are bytes, but must be in a specific encoding (e.g. macOS => UTF-8). On others, they are strings, not bytes (e.g. Windows).

It's implausible to expect Python programmers to correctly handle this, hence fsencode/fsdecode and surrogate escaping.

It's part of the problem of cross platform software. Sometimes you have to go with the least common denominator, in this case byte strings. Do not be smart about it and let the programmer handle the errors. It is unexpected to blow up my software because a user supplied completely valid file name happens to be invalid by Python rules. Yes, fsencode was finally added to work around parts of this problem which is good, but it shouldn't have been there in the first place.

The problem is universal to every time you interface with the operating system. Sometimes it's locale dependent too and you never know beforehand so a program might blow up in the hands of a customer. While the encode/decode process was tiresome at least it gave predictable results.

If you have to work around your runtime in order to give the parameters you want to the underlying syscalls, you are doing it wrong. The premise to move to unicode was a laudable one, but I wish more attention had been paid to how other languages do it, because anything worse than what Perl 5 did more than a decade earlier isn't that great.

Surrogate escaping has been part of Python since 3.2 or so which pretty much solved this.

Python's behaviour is only problematic when all of the below is true:

- Running on Linux and,

- using glibc and,

- glibc's locale files were deleted or the locale is otherwise not configured and,

- you try to read/write non-ASCII file names. (In this scenario Python falls back to the only locale it can process by itself, which is 7-bit ASCII -- the problem is not ASCII, rather the 7-bit restriction. I believe there is a bug open that suggests changing this fallback to UTF-8, which would solve this problem.).

The only time I saw that was with rear which did all of the above on purpose to save like 5 MB on the generated images. In all other cases there is no problem with any kind of file name.

Note that this is a severely broken system configuration where a lot of other stuff falls apart as well.

(Generally speaking I would greatly prefer it if CPython wasn't so reliant on the libc. For example, on a 32-bit system you'll typically find that CPython suffers from y2036 for many date and time operations, including formatting. This would of course add a lot of size to the already big CPython distribution.)