News for hackers. Stuff that matters.
Re:Rethinking the filesystem as global mutable state, the ro... by mrfredward on Tuesday June 11, 2019 @10:45AM (#20155880)
Every piece of hardware on a computer is a piece of global mutable state. There are many good reasons to hide that fact behind an abstraction, but I can't help but think hiding global mutability is better handled at the application level than the OS level, because there are too many cases where the abstraction becomes extremely limiting.

As an example, most people would want to be able to import an image into a word processor regardless of where that image is located (local drive, network drive, floppy disk, etc.). To support that, most end user programs would want to be offered access to the entire filesystem. The moment two applications do this at once, you have all the shared mutable state problems we do now.

Re:Rethinking the filesystem as global mutable state, the ro... by satyenr on Tuesday June 11, 2019 @11:00AM (#20155999)
It doesn't have to be that way. Consider the following scenario:

1. Every file is owned by application.

2. A file-store application is the custodian of files meant to be used by multiple applications.

3. An application that needs to edit a file must take ownership of the file for the duration of the editing.

4. An application that needs to read a file must borrow the files.

5. A file can be borrowed by multiple applications, but owned by only one.

6. Applications can provide the ability for shared ownership, but those would by specialized applications capable of handling and merging simultaneous changes.

This is a straw-man solution, and I am sure multiple problems will have to be solved before such a system can become a reality.

> As an example, most people would want to be able to import an image into a word processor regardless of where that image is located (local drive, network drive, floppy disk, etc.). To support that, most end user programs would want to be offered access to the entire filesystem. The moment two applications do this at once, you have all the shared mutable state problems we do now.

I don't see why the custodian of photos can't be a photo management application. In fact, a filesystem is the lowest common denominator. It is possible to build higher level abstractions like PhotoStore, MusicStore, MovieStore, CodeStore etc. which accommodate and make use of the properties of individual data types to offer an enhanced experience.

Re:Rethinking the filesystem as global mutable state, the ro... by heavenlyblue on Tuesday June 11, 2019 @12:16PM (#20156790)
Your example would fail to accommodate the most basic application of files: logging and reading file logs at the same time.
Re:Rethinking the filesystem as global mutable state, the ro... by satyenr on Tuesday June 11, 2019 @12:20PM (#20156832)
Who says logs have to be written to a file — at least as far as the user is concerned? Logs are a series of events — a LogStore will let you read events as they are appended. And even if you do want to store logs in “files”, the solution proposed here here let’s you borrow them for the owning application.
Re:Rethinking the filesystem as global mutable state, the ro... by heavenlyblue on Tuesday June 11, 2019 @01:47PM (#20157719)
My question goes more like this: “why the hell do you want to re-approach the idea of a filesystem if the new approach can’t even accommodate something as simple as storing log files?”.

It may be time for something completely different. Just stop calling it a filesystem.

Re:Rethinking the filesystem as global mutable state, the ro... by satyenr on Tuesday June 11, 2019 @03:02PM (#20158463)
Again, why do you care about files? Logs have nothing to do with files. Files are just the interface used by filesystem to show the event stream that is logs so you can use file based tools to access the.

I am not calling it a filesystem. Let’s call it a data bank?

Re:Rethinking the filesystem as global mutable state, the ro... by heavenlyblue on Tuesday June 11, 2019 @04:59PM (#20159487)
It already exists in the form of databases.

First that comes to mind is etcd.

Re:Rethinking the filesystem as global mutable state, the ro... by mchristen on Tuesday June 11, 2019 @01:36PM (#20157597)
Where do you plan to store the logs then?
Re:Rethinking the filesystem as global mutable state, the ro... by satyenr on Tuesday June 11, 2019 @03:03PM (#20158467)
On the disk, as usual. Files are, after all, an interface.
Re:Rethinking the filesystem as global mutable state, the ro... by TheCoelacanth on Tuesday June 11, 2019 @01:14PM (#20157396)
That's handled by point 6. You would be able to have shared ownership of the log files, but you wouldn't be able to just write to it willy-nilly. In this case, the specialized application would be trivial because it would just need to ensure that writes are append-only.

> Applications can provide the ability for shared ownership, but those would by specialized applications capable of handling and merging simultaneous changes.

Re:Rethinking the filesystem as global mutable state, the ro... by satyenr on Tuesday June 11, 2019 @03:04PM (#20158483)
Agreed.
Re:Rethinking the filesystem as global mutable state, the ro... by jbreckmckye on Tuesday June 11, 2019 @03:11PM (#20158540)
Surely my logging application would "own" the read-write handle, while my tail application merely "borrowed" a handle for read-only.
Re:Rethinking the filesystem as global mutable state, the ro... by heavenlyblue on Tuesday June 11, 2019 @05:00PM (#20159498)
How do those semantics differ from multi-writers which can all write in transactional blocks?
Re:Rethinking the filesystem as global mutable state, the ro... by pixl97 on Tuesday June 11, 2019 @11:36AM (#20156358)
2 just sounds like reinventing the file system, now with more bugs.
Re:Rethinking the filesystem as global mutable state, the ro... by satyenr on Tuesday June 11, 2019 @11:43AM (#20156430)
No, it is not reinventing the filesystem -- it's more like a file-vault or a file-bank. Anyone looking to operate on the files must either borrow (to read) or take ownership of (to write). The difference is that while a filesystem unlimited access to all files, the app in question is meant to be more sandboxed -- handling only the files it is asked to -- e.g. shared documents that can be edited by multiple apps. Each user (on a multi-user system) will have their file-vault/file-bank and files must be explicitly transferred for shared access.

Finally, as I said, it is a straw-man proposal -- I haven't spent time working out all the kinks.

Re:Rethinking the filesystem as global mutable state, the ro... by zrm on Tuesday June 11, 2019 @12:11PM (#20156734)
> No, it is not reinventing the filesystem -- it's more like a file-vault or a file-bank. Anyone looking to operate on the files must either borrow (to read) or take ownership of (to write).

So flock() and file permissions then?

There are improvements that could be made here -- app-level permissions in addition to user-level permissions for example. But it's still fundamentally a filesystem.

Re:Rethinking the filesystem as global mutable state, the ro... by satyenr on Tuesday June 11, 2019 @12:23PM (#20156875)
All of this can be implemented on top of a file system, of course. What do you think iPadOS does?

We are talking about what the user/application sees and is capable of accessing.

Re:Rethinking the filesystem as global mutable state, the ro... by zrm on Tuesday June 11, 2019 @12:35PM (#20157004)
> We are talking about what the user/application sees and is capable of accessing.

But how is that different than a filesystem?

Suppose we add application-based ACLs to file permissions. Then the app does open("/path/to/file", O_RDONLY) as ever. If the app has permission to the file, it gets the new fd. If it doesn't, it gets EACCES as usual. Or the OS displays a dialog asking whether the app should have permanent or one-time access to that file, and then the call doesn't return until the user chooses one.

I don't see a fundamental change here. The application wouldn't necessarily even have to be modified.

Re:Rethinking the filesystem as global mutable state, the ro... by satyenr on Tuesday June 11, 2019 @12:46PM (#20157124)
By that logic, there is nothing wrong with having a flat memory address space. With access control, the OS can ensure that one application never access another’s memory area. Why do we need isolated virtual address spaces so that application always believes it is the only application running?
Re:Rethinking the filesystem as global mutable state, the ro... by zrm on Tuesday June 11, 2019 @01:05PM (#20157306)
> Why do we need isolated virtual address spaces so that application always believes it is the only application running?

Historically mostly because of swap, so the OS can move a page from memory to disk and then back to a different physical memory location without modifying the application's pointers. On large systems with 32-bit applications it was advantageous because the system may have had more memory than 32-bit pointers can address and then each application can have its own address space. ASLR nowadays.

But filesystems already have the equivalent abstraction. If you run out of space on /dev/sda you can add /dev/sdb, copy /home to it and then mount /dev/sdb1 /home and the application that reads /home/alice/file is blissfully unaware that anything has changed. Heck, half the time you're not even reading from the physical drive, the data is cached in memory and you're really reading it out of the page cache.

Re:Rethinking the filesystem as global mutable state, the ro... by taffer on Tuesday June 11, 2019 @09:24PM (#20160976)
That's interesting. Some time ago I had a similar idea, but with the following differences:

3. Ownership cannot be transferred. A file remains in the same application throughout its lifecycle.

4. An application can give another application read-only access to a namespace containing one or more files.

6. An application can give another application read/write access to a namespace containing one or more files through a transaction mechanism.

Do you see any advantages or disadvantages of both approaches from your point of view?

Re:Rethinking the filesystem as global mutable state, the ro... by satyenr on Wednesday June 12, 2019 @12:08AM (#20161691)
The idea behind ownership transfer is to ensure that an application never sees another application’s namespace. Ownership transfer doesn’t really involve copying files as it does on iPadOS — it is still held at the same location on disk. The original application can just no longer see it as it no longer remains in its namespace. At the end of the operation, the second application may choose to retain or relinquish the file as needed.
Re:Rethinking the filesystem as global mutable state, the ro... by Retra on Tuesday June 11, 2019 @11:22AM (#20156216)
I have photos on my computer, even though I don't use a photo management application. I have compressed files. I have files taken from other machines. I remove applications. I create my own file formats. Applications themselves are files.

Who owns all the unassigned stuff? The file system. Give it a different name and you change nothing.

Re:Rethinking the filesystem as global mutable state, the ro... by satyenr on Tuesday June 11, 2019 @11:37AM (#20156366)
You don't use a photo management application because the file-system exposes photos as files. It is like saying -- in java parlance -- I am perfectly fine using object for everything -- I don't need any other data type. In the hypothetical system I proposed, the photo store would be exposed via an application like Explorer/Finder -- just tailored for viewing photos.

Compressed files are containers for files -- they are nothing on their own.

> Applications themselves are files.

Sure -- they go into ApplicationStore -- which tailored for say, quick application discovery and launch.

> Who owns all the unassigned stuff? T

What exactly is the unassigned stuff? Files taken from other machines? I don't see why they can't be put into an appropriate store based on type.

> I create my own file formats.

I assume you create your own file formats because you develop your own applications? In that case, your application will be the owner of the file in question.

Re:Rethinking the filesystem as global mutable state, the ro... by michaelmrose on Tuesday June 11, 2019 @02:52PM (#20158377)
What if I don't desire a gui application store or desire a different one or switch application stores or move a dir with executable files from one machine to another?

Is the application store the only way to start at app? If I bind a global hotkey to start an app how is that handled?

How about I have an addon for firefox that allows me to bind keys to operations which can include javascript which can itself start applications which can then write to files.

I use this for example to open links in mpv with a singular line in my config file for the addon, tridactyl,

bind V hint -W ! mpv this means show a hint on links and for the given link send the chosen link to mpv, wherein mpv will look at the link and if it can read it will use youtube-dl to download the link to a temp dir and display the video.

Who owns what there?

Do I have to access my photos via a a singular photo manager? I can imagine images accessed in 17 different contexts by 30 different apps does each of them need permission to access each dir which contains images or just generically to access images.

Does this mean that a singular permission would control both access to the browsers cached image of the ycombinator logo on this page and some ones nude selfies?

I just don't think you can cleanly map apps -> files beyond the trivial cases without making something inflexible that sucks to use.

Re:Rethinking the filesystem as global mutable state, the ro... by satyenr on Tuesday June 11, 2019 @03:14PM (#20158570)
> What if I don't desire a gui application store or desire a different one or switch application stores or move a dir with executable files from one machine to another?

What does any of this have to do with a GUI? And for transporting, extract the data into a portable bundle, or take the entire store.

> Is the application store the only way to start at app? If I bind a global hotkey to start an app how is that handled?

Again, what does any of this have to do with how the applications are stored? Can you see the files that make up an iOS app? You can still start them, can’t you? A macOS app is a folder called Something.app, the actual binary lies somewhere inside. Do you typically need to poke inside the .app folder?

> How about I have an addon for firefox that allows me to bind keys to operations which can include javascript which can itself start applications which can then write to files.

As far as the computer is concerned, you are merely starting a process. Again, what does that have to do with how applications are stored. They can be triggered however and wherever they are stored.

> Do I have to access my photos via a a singular photo manager? I can imagine images accessed in 17 different contexts by 30 different apps does each of them need permission to access each dir which contains images or just generically to access images

This is already being handled on iOS and Android. Photos apps is merely the default interface for the PhotoStore. You can access your photos from any app.

> Does this mean that a singular permission would control both access to the browsers cached image of the ycombinator logo on this page and some ones nude selfies?

Namespaces inside individual stores, or separate stores.

Re:Rethinking the filesystem as global mutable state, the ro... by michaelmrose on Tuesday June 11, 2019 @03:54PM (#20158949)
1. At present I can take a text file even not on the PATH put a shebang on it, mark it executable, and execute it because the underlying medium is a user visible hierarchical data store that I interact with directly. It's not clear how the app store application would be associated with it or even know about it or how say a terminal would borrow the application that it didn't know about to run it.

2. An actual filesystem allows you to run things that aren't in an app store bundle

3. Which app owns the files when An app starts an addon which starts a process which runs an app which accesses a file. Does the last app in the chain own the file? This is challenging because plenty of apps could do things based on arguments passed in that involve modifying the filesystem.

4. The way ios and android handle 2 apps accessing the same file is only acceptable if someone has envisioned the way you want this to happen to some degree on both ends. A file picker works on any type of file and an OS that can't have a file picker seems to be objectively worse whereas adding an image picker would be an upgrade. I enjoy using calibre to manage how ebooks are stored and beets for music but neither is bounded by an underlying system designed by others and neither locks said files into said structure or limits access according to it. Its trivial to all out to calibre including via the cli get the full path to a book for example and do something with it.

It seems fairly clear to me that there several layers of filesystem access.

Applications that should never have access to your filesystem because they are malicious. Avoided by installing only from trusted sources to avoid malware in the first place the best possible line of defense. This also makes it possible to get a list of applications that ought to be revoked and communicate this to users.

Running code that should have no or very very controlled access to the filesystem ordinarilylike the Javascript on this page.

Apps that run as the user on their behalf that accesses the filesystem.

You appear to want to pile a layer on top of the last. This appears to only work for the simpler cases and I'm not even clear what the benefits are supposed to be.

Re:Rethinking the filesystem as global mutable state, the ro... by taffer on Tuesday June 11, 2019 @10:06PM (#20161154)
> It's not clear how the app store application would be associated with it or even know about it or how say a terminal would borrow the application that it didn't know about to run it.

Where does the file come from? Suppose you create a file in your terminal application, then it lives inside the terminal and you can use your terminal to run it.

> 2. An actual filesystem allows you to run things that aren't in an app store bundle

Neither a global namespace (file system) nor an appstore are required to execute a program.

> Which app owns the files when An app starts an addon which starts a process which runs an app which accesses a file.

This should be at the discretion of the application developer. This is also the way browsers already work with sandboxed addons.

> This appears to only work for the simpler cases and I'm not even clear what the benefits are supposed to be.

If you don't put everything in the same global namespace, you get more security, cohesion and compatibility. That's why we use VMs, containers, sandboxes, and various user accounts instead of doing everything in a single filesystem using the root user.

Re:Rethinking the filesystem as global mutable state, the ro... by michaelmrose on Wednesday June 12, 2019 @12:01PM (#20166036)
I think constraining many apps access to the file system is a fine idea. I just think think that mapping files to applications very quickly becomes farcical as the abstraction just clearly doesn't match.

It not even optimally secure. Why for example would your image editor need access to all your image files instead of just the ones passed in via a secure system dialog?

In that instance the dialog would be the checkpoint not the some weird file system borrow checker.

Re:Rethinking the filesystem as global mutable state, the ro... by empath75 on Tuesday June 11, 2019 @11:34AM (#20156339)
Sounds like rust’s borrow checker but for files.
Re:Rethinking the filesystem as global mutable state, the ro... by satyenr on Tuesday June 11, 2019 @11:37AM (#20156371)
That is where I got the idea. :-)
Re:Rethinking the filesystem as global mutable state, the ro... by amelius on Tuesday June 11, 2019 @11:56AM (#20156586)
Instead of "ownership transfer" you can also use locks. Read and write locks are already supported on many filesystems.
Re:Rethinking the filesystem as global mutable state, the ro... by satyenr on Tuesday June 11, 2019 @12:25PM (#20156888)
Well, you can also use global variables safely with locks.
Re:Rethinking the filesystem as global mutable state, the ro... by auggierose on Tuesday June 11, 2019 @12:02PM (#20156651)
How much is that different from how it works on iPadOS?
Re:Rethinking the filesystem as global mutable state, the ro... by satyenr on Tuesday June 11, 2019 @12:25PM (#20156898)
It is meant to be more powerful and flexible than what iPad OS does, but somewhat similar.
Re:Rethinking the filesystem as global mutable state, the ro... by LoSboccacc on Tuesday June 11, 2019 @01:30PM (#20157542)
basically the windows document folder
Re:Rethinking the filesystem as global mutable state, the ro... by satyenr on Tuesday June 11, 2019 @03:14PM (#20158578)
Except, you won’t see files, just photos, movies, documents, etc.
Re:Rethinking the filesystem as global mutable state, the ro... by titzer on Tuesday June 11, 2019 @12:01PM (#20156642)
> Every piece of hardware on a computer is a piece of global mutable state.

It's a tenuous analogy. DRAM, disks, caches...effectively the entire memory hierarchy is implemented as message-passing. Flip flops can be well, flipped, but those are super low-level. Almost all the rest of it requires you to send messages. It's kind of ironic that all those messages basically orchestrate the illusion of global mutable state.

Re:Rethinking the filesystem as global mutable state, the ro... by dooglius on Tuesday June 11, 2019 @01:16PM (#20157428)
No, communication is implemented as message-passing (but that is largely irrelevant), but ultimately the end result is changing physical state.
Re:Rethinking the filesystem as global mutable state, the ro... by yomly on Tuesday June 11, 2019 @12:29PM (#20156944)
>It's kind of ironic that all those messages basically orchestrate the illusion of global mutable state

Presumably because for certain things it was easier to reason about when things were mutable...

And now we're taking this facade of mutable state and trying to add a veneer over it to make it seem immutable.

It's circular :)

Re:Rethinking the filesystem as global mutable state, the ro... by BubRoss on Tuesday June 11, 2019 @05:13PM (#20159598)
I think this is a really good point. QNX aligned much closer with this architecture as far as I can tell.
Re:Rethinking the filesystem as global mutable state, the ro... by dooglius on Tuesday June 11, 2019 @12:19PM (#20156827)
Doesn't POSIX already let you lock files? I don't see any reason to make it mandatory though, it not as if locking gives you a security boundary.
Re:Rethinking the filesystem as global mutable state, the ro... by BubRoss on Tuesday June 11, 2019 @05:11PM (#20159586)
Sounds like QNX, which we still haven't managed to get back to.