The offset is only a tiny part of how POSIX is stateful. The very fact that each read or write is associated with a particular fd, therefore with a particular authorization and lock context, is more of an issue at the servers. Even more of an issue is the possibility of still-buffered writes, which POSIX does require be visible to reads on other fds.
> At least in principle, there's no reason that remote filesystems should know or care about file descriptors
Untrue, and please don't try to "correct" others with your own inaccurate information. As I just said, each file descriptor (or file handle in NFS) has its own authorization and lock context, which must be enforced at the server(s) so knowledge of them can't be limited to the client.
> POSIX access bits are literally 15 bits per file. uid and gid are a few bytes.
Also mtime and atime, and xattrs which can add up to kilobytes, but more importantly what the author was really talking about was namespace information rather than per-file metadata. It's a common mistake. Even as someone who writes code to handle both of these separate concerns, I'm not enough of a pedant to whine every time an application programmer gets my domain's terminology wrong.
> the only one of the three POSIX file timestamps - access time - that actually can cause big scalability issues (if left enabled).
Untrue yet again. Mtime can be a problem too, as can st_size and st_blocks. In an architecture where clients issue individual possibly-extending writes directly to one of several data servers for a file but other clients can then query these values through a separate metadata server, that creates a serious aggregation problem. That's why I think the separate ODS/MDS model (as in Lustre) sucks. People resort to it because it makes the namespace issue easier, but it makes metadata issues harder. In the particular use cases where people have to stick with a filesystem instead of switching to an object store, it's a net loss.
The article points out that file descriptors are stateful - they have a current offset associated with them, and I/O syscalls update that offset - then claims that this "stateful model" could be a "scalability bottleneck" with "billions of processes" in parallel filesystems. Except that each of those billions of processes will have its own file descriptor with its own offset! The only case where this could cause contention is if multiple threads in a single process are trying to read from the same file descriptor, which would be a really bad idea in the first place, precisely because file descriptors are stateful. Just open multiple descriptors - or use pread/pwrite, which have existed for a long time. Perhaps the process-wide statefulness of many POSIX APIs is a bad design in a world with threads, but it has nothing to do with the concurrent-file-open benchmark in the article, or really any other performance problems with parallel filesystems.
Anyway, file descriptors are just per-process handles used for communicating with the kernel. At least in principle, there's no reason that remote filesystems should know or care about file descriptors on the client end, unless clients are using file locks (well, except those aren't file-descriptor-based anyway, although they should be).
Later, the article claims:
> While the POSIX style of metadata certainly works, it is very prescriptive and inflexible; for example, the ownership and access permissions for files are often identical within directories containing scientific data (for example, file-per-process checkpoints), but POSIX file systems must track each of these files independently.
[..]
> Supporting the prescriptive POSIX metadata schema at extreme scales is a difficult endeavor; anyone who has tried to ls -l on a directory containing a million files can attest to this.
POSIX access bits are literally 15 bits per file. uid and gid are a few bytes. The overhead of storing these for each file definitely isn't what's making ls -l slow. Perhaps there's some system where time spent checking them all, at access time, is a bottleneck, but I'd be very surprised if modern Linux was such a system; that kind of problem sounds easy to solve with some basic caching.
The article calls out created and modified times as another part of the metadata, thereby amusingly missing the only one of the three POSIX file timestamps - access time - that actually can cause big scalability issues (if left enabled).
Also, apparently the author has not heard of either ACLs, which are in POSIX, or xattrs, which are pseudo-POSIX (multiple systems have roughly compatible implementations based on an old POSIX draft) - both of which try to improve flexibility compared to classic POSIX metadata. There are problems with both of them, but you'd think they'd at least deserve a mention in the list of alternatives.