Re:50 Years in Filesystems: 1984
by kragen on Saturday March 08, 2025 @08:26PM
(#43305295)
This is a very interesting conversation, thank you!
A couple of minor quibbles: FAT32 is not a compatible extension of the older FAT; ext2fs is not a compatible implementation of any old Unix filesystem; having less features does not always make software worse; the BSD LFS research report is from 01993 rather than 01988 https://www.usenix.org/legacy/publications/library/proceedin... (it credits Ousterhout with introducing the idea of LFSs in 01989); several of the other filesystems I listed that I don't think are significantly influenced by HPFS have not in fact disappeared; and Windows 95 does not, to my knowledge, support extended attributes on existing filesystems.
It's probably worth mentioning that, as you imply, ext2fs did eventually add both extended attributes and B-tree directories, just after being quasi-renamed ext4fs. And I do agree that ext2fs was initially intended to be as boring as possible (without suffering the annoyances of the Minix filesystem) rather than innovative. It just failed to be compatible with BSD FFS.
It's a good point that renaming and hardlinking doesn't automatically take into account such extended attributes implemented as a library; that's a plausible reason to implement them in the filesystem. But implementing them in the filesystem doesn't actually help you with respect to copying, unless copying files is an operation the filesystem supports intrinsically, like "reflinks" on some Linux filesystems such as XFS. You still have to modify your file-copying program to copy the extended attributes when that is desired, whether they're provided by the kernel or by a library. Even if the filesystem supports a copy operation, there are typically cases it doesn't handle, such as copying files onto removable media or network filesystems.
Moreover, whether the attributes are implemented in the kernel or in a library, you also have to modify other utility programs to handle the extended attributes if they are to be preserved; for example, backup programs (including things like rsync, zip, and tar), filesystem integrity checking programs like Tripwire, and network file server programs.
File operations that aren't affected by extended attributes wouldn't need to use the extended-attribute library. Depending on what you wanted to use them for, that could be most of them. And it isn't necessary to reimplement the library in multiple different languages; it's sufficient to implement it in C, Rust, or assembly. (All the other languages would need to have a way to call the C library, but that's also true of extended-attribute system calls.)
If you want to use xattrs for ACLs, you can't implement them in a library. But that seems like a backwards way to look at the situation. You don't need extended attributes in the kernel to implement ACLs; you can extend your filesystem to store ACLs in an ACL-specific place. But, if you do have extended attributes in the kernel, that's a candidate way to store ACLs.
Implementing features like xattrs as libraries (rather than in the kernel) has advantages as well as disadvantages. For example, depending on how you do it, you don't need to modify your backup programs. You can support them on all filesystems instead of just some filesystems. And getting or setting them doesn't necessarily incur system-call overhead, though that probably depends on caching.
I don't think VFAT or exFAT does in fact support either extended file attributes or B-tree directories. According to https://eclecticlight.co/2018/01/12/which-file-systems-and-c..., MacOS fakes xattr support on them using the approach I outlined upthread, but instead of suffixing the filename with .ea, it prefixes it with ._. And I think it does it in the kernel rather than in a library, but I don't really know.
BeOS's ability to find files by extended attributes seems like a reasonable justification for doing them in the kernel instead of in a library. Otherwise, though, it seems like an obvious but debatable design choice like case sensitivity, not an innovation.
A couple of minor quibbles: FAT32 is not a compatible extension of the older FAT; ext2fs is not a compatible implementation of any old Unix filesystem; having less features does not always make software worse; the BSD LFS research report is from 01993 rather than 01988 https://www.usenix.org/legacy/publications/library/proceedin... (it credits Ousterhout with introducing the idea of LFSs in 01989); several of the other filesystems I listed that I don't think are significantly influenced by HPFS have not in fact disappeared; and Windows 95 does not, to my knowledge, support extended attributes on existing filesystems.
It's probably worth mentioning that, as you imply, ext2fs did eventually add both extended attributes and B-tree directories, just after being quasi-renamed ext4fs. And I do agree that ext2fs was initially intended to be as boring as possible (without suffering the annoyances of the Minix filesystem) rather than innovative. It just failed to be compatible with BSD FFS.
It's a good point that renaming and hardlinking doesn't automatically take into account such extended attributes implemented as a library; that's a plausible reason to implement them in the filesystem. But implementing them in the filesystem doesn't actually help you with respect to copying, unless copying files is an operation the filesystem supports intrinsically, like "reflinks" on some Linux filesystems such as XFS. You still have to modify your file-copying program to copy the extended attributes when that is desired, whether they're provided by the kernel or by a library. Even if the filesystem supports a copy operation, there are typically cases it doesn't handle, such as copying files onto removable media or network filesystems.
Moreover, whether the attributes are implemented in the kernel or in a library, you also have to modify other utility programs to handle the extended attributes if they are to be preserved; for example, backup programs (including things like rsync, zip, and tar), filesystem integrity checking programs like Tripwire, and network file server programs.
File operations that aren't affected by extended attributes wouldn't need to use the extended-attribute library. Depending on what you wanted to use them for, that could be most of them. And it isn't necessary to reimplement the library in multiple different languages; it's sufficient to implement it in C, Rust, or assembly. (All the other languages would need to have a way to call the C library, but that's also true of extended-attribute system calls.)
If you want to use xattrs for ACLs, you can't implement them in a library. But that seems like a backwards way to look at the situation. You don't need extended attributes in the kernel to implement ACLs; you can extend your filesystem to store ACLs in an ACL-specific place. But, if you do have extended attributes in the kernel, that's a candidate way to store ACLs.
Implementing features like xattrs as libraries (rather than in the kernel) has advantages as well as disadvantages. For example, depending on how you do it, you don't need to modify your backup programs. You can support them on all filesystems instead of just some filesystems. And getting or setting them doesn't necessarily incur system-call overhead, though that probably depends on caching.
I don't think VFAT or exFAT does in fact support either extended file attributes or B-tree directories. According to https://eclecticlight.co/2018/01/12/which-file-systems-and-c..., MacOS fakes xattr support on them using the approach I outlined upthread, but instead of suffixing the filename with .ea, it prefixes it with ._. And I think it does it in the kernel rather than in a library, but I don't really know.
BeOS's ability to find files by extended attributes seems like a reasonable justification for doing them in the kernel instead of in a library. Otherwise, though, it seems like an obvious but debatable design choice like case sensitivity, not an innovation.