News for hackers. Stuff that matters.
Re:The Python Unicode Mess by masklinn on Saturday October 06, 2018 @10:55AM (#18155428)
> Are they? Is the separator 0x2f or / ? Or are you talking about the filename only, not paths?

It's 0xf2. Encode a path to UTF-16 and watch the entire thing burn, with the FS seeing the path you provided end up right before or after the first / (because it encodes as either 0x00f2 or 0xf200 depending on BE versus LE).

Re:The Python Unicode Mess by loeg on Saturday October 06, 2018 @02:55PM (#18156629)
Unix paths are nul-byte-terminated, so UTF-16 generally doesn't make any sense in this context. Valid Unicode paths are encoded as UTF-8 on unix systems. UTF-16 and UTF-32 are invalid ways to encode Unicode paths. (That's not to say no one has tried to do it, just that it doesn't make any sense.)

(As other commenters have pointed out, Unix paths do not require a specific encoding, so robust applications cannot rely on any assumptions about encoding of existing files. But when creating new files, they must not try to encode paths as UTF-16.)

Re:The Python Unicode Mess by masklinn on Sunday October 07, 2018 @06:53AM (#18159963)
> Unix paths are nul-byte-terminated

That's the point. The separator is not "/", because "/" would be a character to encode. The separator is a specific byte, and so is the terminator.

> Valid Unicode paths are encoded as UTF-8 on unix systems.

There is no such thing as "unicode paths" on UNIX systems, valid or invalid.

Re:The Python Unicode Mess by daeken on Saturday October 06, 2018 @12:10PM (#18155812)
Endianness affects order of bytes (0xbeef vs 0xefbe), not nibbles (0xfeeb).
Re:The Python Unicode Mess by BeeOnRope on Saturday October 06, 2018 @01:55PM (#18156360)
The GP's example shows it affecting bytes, not nibbles.