News for hackers. Stuff that matters.
Re:Where is my file's metadata? by nullindividual on Saturday October 28, 2023 @09:33AM (#38049650)
NTFS has Alternate Data Streams[0] as an analog, but rarely leveraged outside of Microsoft’s use…. except for Macs[1] and AFP/SMB on NT, back in the day. Streams[2] from Sysinternals will let you read/write ADS (along with PoSh and other tools).

ADS may also be a security risk[3].

[0] https://learn.microsoft.com/en-us/openspecs/windows_protocol...

[1] https://learn.microsoft.com/en-us/openspecs/windows_protocol...

[2] https://learn.microsoft.com/en-us/sysinternals/downloads/str...

[3] https://blog.netwrix.com/2022/12/16/alternate_data_stream/

Re:Where is my file's metadata? by loeg on Saturday October 28, 2023 @10:52AM (#38050261)
And of course, Linux has support for extattrs and the common Linux filesystems (ext, xfs, btrfs at least) support them.
Re:Where is my file's metadata? by adrian_b on Saturday October 28, 2023 @07:18PM (#38054388)
The first file system with extended file attributes was HPFS, which was introduced in the version 1.2 of the OS/2 operating system, in 1989.

HPFS was followed by Silicon Graphics XFS and MS NTFS, both in 1993.

After XFS became open source, it was the first Linux file system with support for extattrs, which is why it still has its own commands for handling them, besides those that work with any other Linux file system.

Then the rest of the Linux file systems have also adopted this feature, with the main exception of tmpfs, which for many years did not support any extattrs, then it added support only for certain security-related extattrs and only now support for user extattrs has been announced, but with some limitations.

Re:Where is my file's metadata? by js2 on Saturday October 28, 2023 @10:53AM (#38050266)
Discussion seems incomplete without mentioning AppleSingle and AppleDouble which allow the resource fork to be preserved outside of HFS/APFS:

https://en.wikipedia.org/wiki/AppleSingle_and_AppleDouble_fo...

http://fileformats.archiveteam.org/wiki/AppleDouble

Re:Where is my file's metadata? by didntcheck on Saturday October 28, 2023 @11:50AM (#38050760)
>When the Mac was being developed in the early 1980s, the majority of file systems were crude and simple

I get that this is a Mac-focused site, but isn't starting at the personal computer era missing out a big chunk of history? My understanding was that in the early days of mainframes and then minicomputers, records/files consisting of multiple fields, in a system-standard structure, was the norm, and that Unix was somewhat radical in pushing for the minimalist alternative that "a file is just a bag of bytes [in a hierarchical namespace]". I seem to recall this being discussed in the Unix Haters Handbook, in fact. So while the xattr model may not exactly match the "old way", conceptually it was something of a return to form rather than a wholly novel idea

Re:Where is my file's metadata? by somat on Saturday October 28, 2023 @02:01PM (#38051999)
A lot of the reason that people liked unix was not that it was "better" than it's minicomputer operating system peers but that it embraced simple and holistic interfaces. it is easy to learn and fun to use. It is a hackers os. But robust... Not exactly. My favorite quote which describes the unix viewpoint perfectly is.

"We went to lunch afterward, and I remarked to Dennis that easily half the code I was writing in Multics was error recovery code. He said, "We left all that stuff out. If there's an error, we have this routine called panic(), and when it is called, the machine crashes, and you holler down the hall, 'Hey, reboot it.'"

https://multicians.org/unix.html

footnote: Really the real reason unix was popular was that it was available. AT&T was banned from the operating system business. so it was very easy to get a copy(with source). Many university computer departments did just this. and all those newfangled CS students started hacking on and graduated familiar with the system and brought it to industry with them.

Re:Where is my file's metadata? by zeroimpl on Saturday October 28, 2023 @01:04PM (#38051474)
There's an important difference between metadata owned by a file and metadata that a user/program has attached to a file.

Metadata onwed by a file ought to be embedded in the file's data, so that it will be preserved if you copy the file to a different system. This is the case for images, pdf, mp3 etc.

Metadata attached to a file is useful to simplify many 3rd-party programs, but should generally not be preserved if you copy the file. Stuff like display attributes for Finder, quarantine attributes for the virus scanner, etc.

Each of these requirements deserves a different implementation approach, which is why the current state of affairs makes sense to me.

Re:Where is my file's metadata? by adrian_b on Saturday October 28, 2023 @01:38PM (#38051822)
I do not agree.

Any file copy operation should by default make a perfect copy of the source file, without losing any kind of metadata, because that is the meaning of the word. It is not advertised as a partial copy command.

If a complete copy cannot be done, e.g. because the destination file system cannot store some of the metadata, e.g. it has timestamps with a lower precision or it does not store some of the extended attributes, like the Linux tmpfs file system, the copy operation should fail with an explicit error message, like when the destination does not have enough space for all file data.

Any file copy command should have options to strip some or all metadata from a file, when that is desired, but that should not be the default option.

In the past I had unpleasant surprises with the Linux coreutils or with ssh, which lost silently metadata, even with the appropriate command-line options, which should not have been needed, in the case of coreutils because they had a compilation option to ignore the extended attributes and that option had been used in the binary packages available for certain Linux distributions.

When copying over the network, I always use rsync over ssh, because it copies reliably all metadata, even between different operating systems and file systems, e.g. Linux, FreeBSD and Windows.

Re:Where is my file's metadata? by zeroimpl on Saturday October 28, 2023 @02:10PM (#38052057)
It's a matter of defaults, and no choice of default will work for everybody.

I agree that "rsync --archive" ought to preserve extended attributes, same with creating a tarball. But when copying a file, as opposed to creating a backup/archive, I don't think extended attributes should be included by default.

And yes I've also had to compile custom versions of rsync at one point to get a version which understood extended attributes on Mac, but I think that's mostly standardized now.

EDIT: Also most programs don't even preserve permissions/ACLs on copy, so not sure why they'd preserve attributes by default.

Re:Where is my file's metadata? by adrian_b on Saturday October 28, 2023 @07:02PM (#38054280)
Most copy programs do not preserve much or all metadata by default because this was the original behavior of the UNIX cp command, which copied only the file data, but none of the metadata, except the file name, when it was not specified for the destination.

The first time when I have used cp, more than thirty years ago, it was a shock for me to discover that this is the default behavior, because it is not something that I can associate with a command named "copy".

Since that day and until today, I always alias the various copy commands like cp or rsync to include all the necessary options for making perfect copies.

I have never needed to use them with different options, which would make imperfect copies, but there have been countless situations when my usage would have been broken by lossy copies.

Re:Where is my file's metadata? by pilgrim0 on Saturday October 28, 2023 @03:30PM (#38052674)
I think the concept of a file as a bag of bytes is a leaky abstraction. That model is fit within the context of file system implementations, which are concerned with allocation, but unfit for high level file usage, which is concerned with interpretation. Anyone who has written code for handling file uploads knows that identifying a file format happens on a best effort basis, and it’s never 100% guaranteed to work. Even attempting to check if a file is a “plain text” is not guaranteed to work at all times. File extensions cannot be relied upon, and trusting a magic number prefix is also not good, as it can be easily tampered. Hence the bag of bytes abstraction forces the development of context-specific conventions, thus exploding the complexity in the transactional information space. If we account for the Web and APIs, HTML and JSON are probably the most prevalent file formats. Imagine attempting to develop for such ecosystems using files without embedded metadata, e.g a separate payload for <head> and tag attributes — it doesn’t make any sense. My point is: metadata _is_ data, and I disagree completely with the author in regard to side-channels being “where metadata is supposed to be”. Perhaps for _file-system specific_ metadata, sure, but for file-content metadata, hell no. In the end, because of this abstraction, the trust level upon a file’s metadata after the file is transferred is practically zero. I think that consistently structured data _objects_ with embedded and verifiable cryptographic signatures would be a better fit for the interconnected world. The fact we rely so much on webpages and their associated internet domains as a means of structured context and provenance hints at the importance of objects and signatures.