News for hackers. Stuff that matters.
Re:Rethinking the filesystem as global mutable state, the ro... by JoeAltmaier on Tuesday June 11, 2019 @11:45AM (#20156451)
I've thought for years that 'file systems' are an abomination. We use a cobbled-together schema of parent-directory, filename, some random date/time stamps and maybe a three-letter extension. Why? Because we inheirited that from some DOS days.

Why not a collection of immutable UUID-labelled resources with an arbitrary schema of attributes? Like a relational database or some such.

Overkill? Doesn't every major app have to do this itself, in some container (document/mail folder/image definition/contact list and on and on)?

Designers of operating systems have made no effort to capture this kind of service and provide it as a fundamental OS feature. And OSs are exactly where this belongs - where you can carefully get it right and everybody writing apps can depend on it.

Re:Rethinking the filesystem as global mutable state, the ro... by liara_k on Tuesday June 11, 2019 @01:21PM (#20157468)
I've had very similar ideas, and think this is really critical missing feature that's holding back a lot of progress.

IMHO, what's needed is something like a document database (in that it allows arbitrary schema) but even more importantly, it needs to have some system-level indexing. Right now, applications that deal with large amounts of small pieces of data each roll their own, e.g., embedded database, because (1) most filesystems aren't capable of adequate performance with very large numbers of small objects and (2) you can't maintain an app-independent index anyway, so... These applications then can't interact with each other without some app-specific API, which is often costly to develop against (at least, compared with a common filesystem API) and with only the capabilities that the app developer sees fit to give you. Which they often have very little incentive to do--their real incentives are usually to keep things proprietary and customers captive.

The absence of these two capabilities makes "the unix way" where "everything is a file" an impossible data model for these types of applications.

IMHO, the right data model is probably some combination of user (UUIDs or strings doesn't matter) and content-hash indexing with versioning and conflict resolution similar to Git (and CouchDB).

Re:Rethinking the filesystem as global mutable state, the ro... by AnIdiotOnTheNet on Tuesday June 11, 2019 @12:45PM (#20157101)
> Designers of operating systems have made no effort to capture this kind of service and provide it as a fundamental OS feature.

Actually they've made several. See BeOS and WinFS, for starters, though the idea is even older than that.

Re:Rethinking the filesystem as global mutable state, the ro... by zrm on Tuesday June 11, 2019 @02:05PM (#20157918)
> Doesn't every major app have to do this itself, in some container (document/mail folder/image definition/contact list and on and on)?

It does, but it's also something that can be provided by a library as easily as the OS. So e.g. Firefox uses sqlite, but it can use a portable library for that, and then it isn't different on Windows than Linux or Mac.

And then the application gets to choose something with an appropriate level of complexity. Sometimes all you need is Maildir or xattr(7) or JSON, sometimes you need a full on SQL database. One size fits all rarely does in practice.

Re:Rethinking the filesystem as global mutable state, the ro... by JoeAltmaier on Tuesday June 11, 2019 @03:35PM (#20158779)
Except, the results are opaque. I can't operate on the data without using their tools, which might not be what I want.

Imagine if the OS let you browse all the app data, see all the relations (that you were authorized to see), and write tools to operate over it. That would be a most Open system, compared to everything we have which is essentially walled off to us.

Re:Rethinking the filesystem as global mutable state, the ro... by zrm on Tuesday June 11, 2019 @03:57PM (#20158974)
It's not really opaque as much as it is application-specific. But then how do you fix that? If you give applications the ability to tag things then one mail program uses a tag called "unread mail" and another uses a tag called "message read" with the opposite value and another uses a tag called "mail flags" with a bitmask where one of the bits is whether the message has been read or not etc. Smells like Windows registry.

Either you somehow enforce a high degree of uniformity, which implies a pretty serious lack of flexibility, or everybody gets to make their own decisions and then everybody makes different ones. And the second one seems better as long as the individualized thing they're doing is sufficiently well documented.

Re:Rethinking the filesystem as global mutable state, the ro... by JoeAltmaier on Tuesday June 11, 2019 @04:19PM (#20159157)
The alternative, right now, is nobody can do anything at all like this. There's not even the opportunity to create conventions in apps.

Opening up application data to OS tools, opens up a whole new world of opportunities (and hazards) for app developers.

Re:Rethinking the filesystem as global mutable state, the ro... by syn0byte on Tuesday June 11, 2019 @01:38PM (#20157613)
IBM's i Series, modern heir to OS/400 doesn't have a "filesystem" in the traditional sense (but tons of tools to emulate and approximate it). Everything ends up in DB2.

I've spent time crawling around the SCSI/SAS/Storage of systems like that. Wrote dirty hacks for end-user clients to cope with incompatibilities. EG multiple versions with the same filename are totally fine. Filenames needn't and aren't unique in such systems, just another col in the DB. How do you tell a Windows ftp client which of the 20 different "report.txt" files it should download? (Filezilla actually has 'VMS' options for this, its more common than you think.)

If all you ever deal with is relatively simple, well defined, CRUD type applications on entirely abstract systems in some remote infrastructure, its easy to think that's how all development should look. It doesn't and it shouldn't.

Re:Rethinking the filesystem as global mutable state, the ro... by JoeAltmaier on Tuesday June 11, 2019 @03:32PM (#20158756)
By attributes. Which one did you want? The latest? The one created by app X? The one you printed yesterday?

The fact is, you can do these queries if you allow flexible attributes. With a fixed file system (almost all file systems) you cant ask. In practice, you've already overwritten the one you want, or one app conflicted with another, or you just can't do you want (have two versions of an app installed at the same time).