News for hackers. Stuff that matters.
Re:Unicode is harder than you think by sedatk on Tuesday July 25, 2023 @06:15PM (#36870391)
Certainly not true for Windows. Windows uses UTF-16; e.g. it uses proper surrogate pairs.

https://learn.microsoft.com/en-us/windows/win32/intl/surroga...

Re:Unicode is harder than you think by skitter on Tuesday July 25, 2023 @06:41PM (#36870718)
That would be great, but that article is about recommendations for applications running on Windows, not about what valid file names applications may encounter.
Re:Unicode is harder than you think by sedatk on Tuesday July 25, 2023 @06:46PM (#36870786)
No, I mean Windows API honors UTF-16 surrogate pairs, and processes them correctly. It doesn't produce invalid UTF-16 strings either. Apps may not support UTF-16 properly, and that's not on Windows, is it?

NTFS, on the other hand, has no dictated format for filename encoding. It just stores raw bytes as filenames, so anything can be a filename on NTFS, including invalid strings if the caller decides to do so. That's not on Windows either, otherwise, we should add Linux to the list too as ext4 and most other file systems also don't care about filename encoding.

Re:Unicode is harder than you think by chungy on Tuesday July 25, 2023 @07:27PM (#36871134)
Windows was designed around UCS-2, while it can be upwards compatible with storing UTF-16 data, the Windows APIs don't stop you from inserting almost any sequence of 16-bit words as file names, including unpaired surrogates (because Windows doesn't actually care about the contents of the 16-bit words).

It just happens that most Windows programs will treat it as UTF-16 and it'll mostly be fine. Until you run into the edge case where you do have file names that aren't UTF-16.

This is basically a result of Windows's file API being fixed in 1993, a good few years before Unicode decided that 16 bits wasn't enough for everyone, and hence a good few years before UTF-16 existed.

Re:Unicode is harder than you think by dahfizz on Tuesday July 25, 2023 @07:33PM (#36871181)
Why is that a problem? That's how unix does it as well.

A file path is just a sequence of bytes. Don't try to validate it in any sort of encoding, it's just bytes that the OS uses to identify a file.

Re:Unicode is harder than you think by chungy on Tuesday July 25, 2023 @07:38PM (#36871227)
The problem is that if the operating system allows sequences that are not valid UTF-16, then you cannot just state "it uses UTF-16": anything that is expecting UTF-16 and only UTF-16 is going to break on valid file names.

"That's how unix does it as well" is the exact same thing. Most people these days expect file names in UTF-8, but not all file systems restrict the valid character sequences to be UTF-8 compliant. If you treat everything as if it must be UTF-8, your application may just break when it encounters a name that isn't UTF-8. (This detail gets messy because there are configurations of, eg, ext4 and ZFS that limit valid file names to be valid UTF-8.)

Re:Unicode is harder than you think by OkayPhysicist on Tuesday July 25, 2023 @07:51PM (#36871319)
Does this actually matter? Who is out there, in the wild, creating file paths that use incredibly cursed random sequences of bytes? Maybe it's just me, but I don't really see the need to accommodate people who do stupid things to see what breaks.
Re:Unicode is harder than you think by chrismorgan on Tuesday July 25, 2023 @08:07PM (#36871431)
On Windows, it doesn’t matter much because, although the file system doesn’t validate Unicode, the usual system calls that work with file names do, so it’s very rare to encounter non-Unicode file names.

On Linux, it’s not as rare as you might expect to end up with a few non-UTF–8 file names, though they’ll normally be on things you won’t often touch directly, so things like desktop software can reasonably only support UTF-8 paths.

Re:Unicode is harder than you think by OkayPhysicist on Tuesday July 25, 2023 @08:16PM (#36871499)
Like what?

I'm getting strong XKCD "Workflow"[0] vibes out of the idea that someone would have unintelligible paths on their system

[0] https://xkcd.com/1172/

Re:Unicode is harder than you think by chungy on Tuesday July 25, 2023 @08:32PM (#36871628)
I'll just say: consider yourself very lucky to not having encountered it. (Especially if you've spent any time downloading random zip archives from the internet...)

I have quite a number of times. There's no one "thing"; it's usually some legacy app that write these names. And when you get a program that refuses to open any non-UTF16 compliant file name, you'll have a time of grief.

Re:Unicode is harder than you think by derobert on Wednesday July 26, 2023 @01:11AM (#36873371)
Some unarchivers (especially ones which aren't Unix natives, like say rar) seem to love to do it. They're just buggy, of course.

Beyond that, before the mid-2000s, it was common to use non-UTF-8 locales on Linux. So I'm sure I still have ISO-8859-1/—15 encoded file names somewhere, especially in archived data. They're not always trivial to rename either, because there might be references to them by name. (Or, in odd cases, you can't convert the name to UTF-8 because you hit a filename length limit, since UTF-8 is more bytes).

I believe wanting to access data from 20 years ago is a perfectly reasonable use case.

It's not so bad if a program can't display the file name right, as long as it doesn't crash with an exception or refuse to open the file. Unix file names have been defined as arbitrary sequences of octects except / and NUL for 30+ years.

Re:Unicode is harder than you think by dataflow on Tuesday July 25, 2023 @08:05PM (#36871410)
> Don't try to validate it in any sort of encoding

That works fine until it becomes literally impossible when your language/library/framework forces you into picking an encoding.

Even Microsoft's own APIs that add "UTF-8 support" choke when they encounter file names with invalid UTF-16 sequences.

Re:Unicode is harder than you think by sedatk on Tuesday July 25, 2023 @08:14PM (#36871487)
Windows used to use UCS-2 because UTF-16 hadn't been invented yet. Any Windows after Windows 2000 supports UTF-16.

You can't just dismiss the whole operating system API space just because NTFS doesn't enforce filename encoding. Yes, you might encounter file names with invalid UTF sequences because some app might be allowing them to be created. That can also happen in Linux too. Then add Linux, and probably others to the list. Then, what's the use for this list?

Re:Unicode is harder than you think by kzrdude on Tuesday July 25, 2023 @08:56PM (#36871807)
On Linux it's at least explicit that file paths can be any string of bytes (any or "no" encoding)
Re:Unicode is harder than you think by sedatk on Wednesday July 26, 2023 @05:56PM (#36885684)
I don't understand why this gets brought up. You still have to assume the filenames are encoded to something in order to display them, otherwise there would be no way to display a filename other than in hexadecimal. Windows always uses UTF-16 while Linux uses the locale (which defaults to UTF-8). It's not like working with filenames on Linux is somewhat easier or more straightforward, it's certainly not.