News for hackers. Stuff that matters.
Re:The History and rationale of the Python 3 Unicode model f... by cabirum on Wednesday December 14, 2022 @12:22PM (#33986655)
Filenames come from multiple filesystems (and not just filesystems!), with different encodings and such. Both ntfs and ext3 can use arbitrary bytes as names, not representable in utf8. (Contrary to popular belief, ntfs works with any byte sequences except 0x0000, and may contain invalid utf16 code points)

The only sane way to deal with them all is to use bytes and bytes only (Or use a sane language with encoding-agnostic string type; see: golang) Other solutions introduce loss of data.

Re:The History and rationale of the Python 3 Unicode model f... by deathanatos on Wednesday December 14, 2022 @01:26PM (#33987608)
> The only sane way to deal with them all is to use bytes and bytes only

It's not really, though. Yes, if you want to round-trip the name of the file and be fully general over the bad situation provided by the OS. But as soon as you want to display that filename, or really do much of anything with it, you're back into the world of hurt the article describes, and you need some super complicated type or logic that can convey to the user "hey, this file's name is some random undecodable garbage".

The sane state would have been "file names are text", but filenames/Unix predates software engineering having a good discipline around text encoding, and it predates Unicode. I'd go as far as to argue that "file names are text and \n is disallowed" is the sane state, as multi-line filenames also just cause all sorts of unnecessary complexity.

The problem, of course, is that most OS enforce neither constraint, which is what leads to things like the struggle Python has gone through, or the Path type in Rust, and the unrealistic nature of trying to move from the present state to such a sane state. (Linux is loathe to break compatibility, and I bet Windows would be similar, here. There are a number of arguments for that, but in the meantime, it continues to be that file names are ridiculously complicated if handled fully correctly.

Or, you just decode them as Unicode (UTF-8/-16 depending on OS) and give up if the user is crazy. 99.99% of the time that'll work just fine.)

> Or use a sane language with encoding-agnostic string type; see: golang

Of course, this might be the root of our disagreement. Types that permit the representation of invalid values are not desirable, to me. Especially in a core type, like strings.

Re:The History and rationale of the Python 3 Unicode model f... by saurik on Wednesday December 14, 2022 @01:40PM (#33987894)
> Or, you just decode them as Unicode (UTF-8/-16 depending on OS) and give up if the user is crazy. 99.99% of the time that'll work just fine.)

You have to be really careful about the way in which you give up here, as the user isn't necessarily in control of the files on disk: it might be an attacker. If I can hide files in folders from various parts of a Python program -- as was possible with early versions of Python 3, which simply skipped such files -- or cause corruption in how the filenames are round-tripped during a copy (allowing me to bypass a filter in one place but then write to a file with a different critical name in another place) I gain a lot of flexibility with my exploits.

Re:The History and rationale of the Python 3 Unicode model f... by deathanatos on Wednesday December 14, 2022 @03:30PM (#33990027)
Yeah, and that's a problem. But again, Python is faced with bad choices because the interface exposed by OSes for file systems is basically broken.

(But, that might also fall into the 1% case where you'd need to pay attention.)

Perhaps you might argue for a dedicated type like Rust's Path, but that too has issues, and I don't think Python had the "design freedom" to go there. (Although it does have pathlib, nowadays, so I wonder how it handles this; but it would also need to duplicate a lot of the functionality in os…)

Re:The History and rationale of the Python 3 Unicode model f... by throw0101a on Wednesday December 14, 2022 @02:20PM (#33988652)
> It's not really, though. Yes, if you want to round-trip the name of the file and be fully general over the bad situation provided by the OS. But as soon as you want to display that filename, or really do much of anything with it

IMHO, that is mostly not the file system's problem. Bits going into and out of the file system should act as deterministically as possible.

If you want to start being 'clever' about these problems, do it in userland.

Re:The History and rationale of the Python 3 Unicode model f... by deathanatos on Wednesday December 14, 2022 @03:26PM (#33989953)
And the point is that this line of thinking introduces a metric crap ton of unnecessary complexity downstream.

Yeah, an FS could say that¹. Alternatively, it could say that names are intended to be read by humans, and that names should be a single line, to support things that people reasonable expect to be able to do with a file system, like being able to list the files on the filesystem, without whatever is doing the listing needing to decide if it is going to just say "I can't display raw bytes, obviously, so here's a best-effort of ���.txt" or if it has to invent some sort of grammar to encode/escape bytes into text.

I.e., a file system is meant to solve a problem, which is the storage and organization of data, and from that, that names are meant to used by humans is one of the requirements of both the OS & the filesystem. (The OS, as it presents the file hierarchy to userspace, e.g., in the VFS, and the filesystem, as it of course needs to store the data.)

By your logic, we should permit nul and '/' in file names, too.

¹and, I mean, most do. Which is why we're in the situation we're in, and why languages are faced with a barrel of bad choices, and why probably 90% of shell scripts fail to properly handle file names. Is this what we want?

Re:The History and rationale of the Python 3 Unicode model f... by gary_0 on Wednesday December 14, 2022 @03:37PM (#33990142)
Nobody's keyboard even has a "null" key, so in order to specify garbage-bytes filenames you have to resort to programmatic shenanigans anyways, which means you either have a bug or a malicious programmer. In which case you're going to have to break out specialized tools to fix your broken filesystem--or if you're lucky, you can just delete the directory and start over.

Otherwise, you just want filenames you can pipe straight to text in front of the user, who in any case is going to have no idea what to do with a file named ���.txt.

Re:The History and rationale of the Python 3 Unicode model f... by comex on Thursday December 15, 2022 @12:10AM (#33995493)
> why probably 90% of shell scripts fail to properly handle file names. Is this what we want?

If you want to fix shell scripts, you need to ban not just newlines but also spaces. Well-written shell scripts handle spaces properly, but many don't, and Makefiles cannot.

Re:The History and rationale of the Python 3 Unicode model f... by CorrectHorseBat on Wednesday December 14, 2022 @03:01PM (#33989431)
ZFS even has a utf8only flag that disallows anything but utf-8. Sadly it hasn't caught on with other filesystems.
Re:The History and rationale of the Python 3 Unicode model f... by ilyt on Wednesday December 14, 2022 @04:52PM (#33991128)
Kinda waste of CPU to test for that tbh. Broken app won't work any better coz of it
Re:The History and rationale of the Python 3 Unicode model f... by ilyt on Wednesday December 14, 2022 @04:51PM (#33991112)
> It's not really, though. Yes, if you want to round-trip the name of the file and be fully general over the bad situation provided by the OS. But as soon as you want to display that filename, or really do much of anything with it, you're back into the world of hurt the article describes, and you need some super complicated type or logic that can convey to the user "hey, this file's name is some random undecodable garbage".

It really is. Once you are using byte type you can handle displaying byte type to user as you see fit but that's not a decision stdlib should make for you. Not only that, if any file-related function operates on bytes that by definition it doesn't care about FS semantics, you're also not wasting any CPU for useless conversions to string.

The blob of bytes acts as opaque token for file location, as it should. If you want to display it to users you have to quote it anyway, because you can have perfectly valid utf8 file name that contains newline or some special utf8 characters that will fuck up how it is displayed and might even in malicious way

>> Or use a sane language with encoding-agnostic string type; see: golang

>Of course, this might be the root of our disagreement. Types that permit the representation of invalid values are not desirable, to me. Especially in a core type, like strings.

string is defined as "read-only slice of bytes", not "UTF8 slice of bytes". That is mostly so conversion between is free. If you want to pay the cost of validation you can, but are not forced to.

The functions operating on them, if relevant to the function's functionality, will operate in UTF8.

It might sound bad theory and it did to me at first but in practice it's probably the least problems I had out of all languages.

Re:The History and rationale of the Python 3 Unicode model f... by chungy on Wednesday December 14, 2022 @12:36PM (#33986842)
NTFS disallows 0x002F as well; it is possible to use the entire set even from Windows (installing Interix, for instance). Windows NT was designed around UCS-2, the file system didn't have to do much but be a dumb bag of bytes for file names, character interpretation was on upper layers. You can mostly pretend it's UTF-16, until it's not.

Something fun on file systems: ZFS lets you set "utf8only=on" which will guarantee no file names will ever be possible that are not valid UTF-8. I use it on all my systems.

Re:The History and rationale of the Python 3 Unicode model f... by jcranmer on Wednesday December 14, 2022 @03:29PM (#33990003)
> Something fun on file systems: ZFS lets you set "utf8only=on" which will guarantee no file names will ever be possible that are not valid UTF-8. I use it on all my systems.

This is the sort of option that should be available for all filesystems, and distros should default new installs to turning it on.

Re:The History and rationale of the Python 3 Unicode model f... by deckard1 on Wednesday December 14, 2022 @04:10PM (#33990614)
if we have a time machine then yes. In fact, it shouldn't even be an option. UTF-8 period. Always.

But alas, we do not and someone will plug in a memory card or USB drive with some FAT formatted drive and your nice universe crumbles before you.

Re:The History and rationale of the Python 3 Unicode model f... by chungy on Wednesday December 14, 2022 @05:06PM (#33991292)
This is why "convmv" exists :) and also why my /tmp is a normal tmpfs where file names can be any byte sequence. I can, for instance, extract random archives there, use convmv to make them UTF-8 (if not already), and then it can go on my ZFS file systems.
Re:The History and rationale of the Python 3 Unicode model f... by jcranmer on Wednesday December 14, 2022 @05:25PM (#33991536)
Handling non-UTF-8 pathnames via a filesystem mount option is completely viable. Hell, I'd go even further and suggest that the kernel itself reject non-UTF-8 pathnames entirely, handling filesystems with the non-UTF-8-pathname option enabled via a translation layer that converts \x00-\xff to \u0000-\u00ff (and vice versa).

The lesson of text is clear: text needs to have a defined charset. Filesystem names are absolutely textual (listing a directory for human display is one of the most common operations). If it's not UTF-8, then what is it? How is the application supposed to figure it out? (Automatic encoding detection is flaky in the best of times, but with the lengths of typical filenames, it is almost impossible to be reliable). The reality is that most applications already assume that filesystem names are UTF-8, and non-UTF-8 names break them.

We are at a stage in Unicode penetration that I think it is reasonable to treat a filesystem that has non-UTF-8 filenames as a problem that needs to be solved with some kind of repair tool like fsck, not something that every application is supposed to worry about.

Re:The History and rationale of the Python 3 Unicode model f... by lmm on Wednesday December 14, 2022 @05:45PM (#33991817)
Please don't unless you've thoroughly tested in multiple cultures. Unicode is broken for Japanese; on some systems it's possible to change a setting to make it broken for Chinese instead, but the only non-theoretical way to make a program that can list a directory full of Japanese filenames and a directory full of Chinese filenames correctly is to have that program be encoding-aware.
Re:The History and rationale of the Python 3 Unicode model f... by lidavidm on Wednesday December 14, 2022 @06:09PM (#33992146)
I'm curious, what does 'broken' mean? Is this about the CJK merger, or something else?
Re:The History and rationale of the Python 3 Unicode model f... by lmm on Wednesday December 14, 2022 @06:24PM (#33992334)
Yes, CJK unification. Characters will be rendered differently from how they should be (e.g. with missing or extra strokes).
Re:The History and rationale of the Python 3 Unicode model f... by hnfong on Wednesday December 14, 2022 @10:37PM (#33994859)
Why is that a filesystem issue or a "utf8only=on" issue? If the renderer doesn't have sufficient context, it will pick one glyph to render anyway, and it could be "wrong".

PS: unless your take is to say that we should just assume everything is ascii and don't ever render codepoints > 127?

Re:The History and rationale of the Python 3 Unicode model f... by lmm on Wednesday December 14, 2022 @10:54PM (#33995009)
If the filesystem can't allow parts to be in different encodings then it can't ever possibly work right. With current filesystems at least I can do "ls ~/chinese-files; LANG=ja_JP.SJIS ls ~/japanese-files". Which is not a great way of working, but it's better than a utf8only filesystem where it's physically impossible to make even that much work.
Re:The History and rationale of the Python 3 Unicode model f... by chungy on Thursday December 15, 2022 @12:18AM (#33995548)
If you run a ZFS file system where this is a huge deal to you, then don't set utf8only. If it's set on a parent dataset, then explicitly do utf8only=off.
Re:The History and rationale of the Python 3 Unicode model f... by russdill on Wednesday December 14, 2022 @01:49PM (#33988073)
Even using bytes can lead to loss of data when trying to copy from one file-system to another or dealing with operating systems that treat filenames differently when different system calls are used.
Re:The History and rationale of the Python 3 Unicode model f... by gumby on Wednesday December 14, 2022 @02:44PM (#33989088)
I have used filesystems that permitted a character represented as 0x0. Good way to attack a C program.
Re:The History and rationale of the Python 3 Unicode model f... by GeorgeTirebiter on Wednesday December 14, 2022 @04:33PM (#33990914)
I sometimes think the C decision to encode strings as non-zero bytes was one of the worst.

The other candidate is the atrocious operator precedence. OK, we now love parens to 'fix' this -- might as well use lisp... ;-)

Re:The History and rationale of the Python 3 Unicode model f... by ilyt on Wednesday December 14, 2022 @04:56PM (#33991172)
It was unequivocally one of the worst decisions in programming. I wouldn't be surprised if costs of dealing with it would be in the billions now. It occasionally saved a byte on devices with little memory which kinda made sense at a time, but not anymore
Re:The History and rationale of the Python 3 Unicode model f... by stabbles on Wednesday December 14, 2022 @05:25PM (#33991537)
It makes perfect sense when writing and reading strings from disk, since you don’t have to worry where and how to store the integer for the length (#bytes & endianness)
Re:The History and rationale of the Python 3 Unicode model f... by optionalsquid on Wednesday December 14, 2022 @06:19PM (#33992281)
Wouldn't endianness be a concern regardless, unless you are literally only reading/writing (null terminated) strings? Which I would imagine to be a much rarer case than writing a mix of data-types or just plain text.
Re:The History and rationale of the Python 3 Unicode model f... by gumby on Thursday December 15, 2022 @12:11AM (#33995499)
Many systems used this convention; c was merely one of them. So it was a “choice” more than an”decision”.

The Mesa guys at PARC did a study back in the late 70s and, statistically, length-marked strings are faster than delimited ones.

Delimited strings have some advantages: they save the space needed to store the length, at the cost of an extra byte (not worth the cost these days); and some substrings can be assembled simply by pointing into the string (of course counted strings can share storage). But at a terrible cost.

But remember that the PDP-7 was less powerful than some AVRs. We use more powerful chips to scan keyboards these days.

Re:The History and rationale of the Python 3 Unicode model f... by adgjlsfhk1 on Thursday December 15, 2022 @11:40PM (#34010339)
You could pretty easily make it be very low overhead. Store the length of 128 or fewere byte strings with 1 byte for the length (starting with a zero), and if the first bit is a 1, then the string length takes 4 bytes (or use extra bytes with the same format).