News for hackers. Stuff that matters.
Re:Unicode is harder than you think by OkayPhysicist on Tuesday July 25, 2023 @07:51PM (#36871319)
Does this actually matter? Who is out there, in the wild, creating file paths that use incredibly cursed random sequences of bytes? Maybe it's just me, but I don't really see the need to accommodate people who do stupid things to see what breaks.
Re:Unicode is harder than you think by chrismorgan on Tuesday July 25, 2023 @08:07PM (#36871431)
On Windows, it doesn’t matter much because, although the file system doesn’t validate Unicode, the usual system calls that work with file names do, so it’s very rare to encounter non-Unicode file names.

On Linux, it’s not as rare as you might expect to end up with a few non-UTF–8 file names, though they’ll normally be on things you won’t often touch directly, so things like desktop software can reasonably only support UTF-8 paths.

Re:Unicode is harder than you think by OkayPhysicist on Tuesday July 25, 2023 @08:16PM (#36871499)
Like what?

I'm getting strong XKCD "Workflow"[0] vibes out of the idea that someone would have unintelligible paths on their system

[0] https://xkcd.com/1172/

Re:Unicode is harder than you think by chungy on Tuesday July 25, 2023 @08:32PM (#36871628)
I'll just say: consider yourself very lucky to not having encountered it. (Especially if you've spent any time downloading random zip archives from the internet...)

I have quite a number of times. There's no one "thing"; it's usually some legacy app that write these names. And when you get a program that refuses to open any non-UTF16 compliant file name, you'll have a time of grief.

Re:Unicode is harder than you think by derobert on Wednesday July 26, 2023 @01:11AM (#36873371)
Some unarchivers (especially ones which aren't Unix natives, like say rar) seem to love to do it. They're just buggy, of course.

Beyond that, before the mid-2000s, it was common to use non-UTF-8 locales on Linux. So I'm sure I still have ISO-8859-1/—15 encoded file names somewhere, especially in archived data. They're not always trivial to rename either, because there might be references to them by name. (Or, in odd cases, you can't convert the name to UTF-8 because you hit a filename length limit, since UTF-8 is more bytes).

I believe wanting to access data from 20 years ago is a perfectly reasonable use case.

It's not so bad if a program can't display the file name right, as long as it doesn't crash with an exception or refuse to open the file. Unix file names have been defined as arbitrary sequences of octects except / and NUL for 30+ years.