I feel like it should be possible to say "I want to reserve access to this resource when something important happens on it". Or just "I want to reserve access to this resource". Hopefully eliminates TOCTOU races. I guess then there have to be different modes to balance scalability/flexibility.

When people complain about antivirus and endpoint security applications slowing their computer down, this is what they're talking about. This leads to user-visible delays (which, to them, seem to be random) as the reserver does what it wants right after the user has done something to the file. Many (most? ALL?) apparent Explorer stalls are really Windows Defender locking and scanning the file right after you did something to it.

Yeah good point. I don't want write stalls because of subscription requests. For things like filesystem watching (for compilers), it'd be enough to have a form of snapshot checkpointing. (There were these FS events within 10ms [..]) / (Then the file contents was XXX) / ... (Then there were these FS events) / and so on.

Its possible you'd end up in situations where the OS would need to retain old file contents if there's both a pending read from a subscriber and subsequent writes. But in most cases, this is fine.

Going back to my original comment, this is the sort of reason I'd like to try this in my own OS kernel rather than try and solve all these problems immediately in linux. It'll give me a chance to try out a few different solutions and see if I can find something that works well.

It should be possibly to speculatively execute code that's accessing resources and only allowing transactions on system resources to commit successfully if other transactions haven't invalidated them.

Meaning a process could start loading and reading while queuing writes and only writes would block while another process is locking access to the executable to scan it. If the scan succeeds the writes are committed and if it fails it could revoke execution privileges and all the work is undone.

Lot of hand waving there, but when you're not committed to existing file system semantics there is a lot of room for innovation.

You can definitely do that, it just means that userspace can now stall access to things unless you're very careful :) Like, one model (often used by EDR, which is why everyone hates them) is that every operation must be "authorized" so it doesn't go through until some evaluation happens. But this means the kernel has to ask a program for every operation, which is slow.

What's EDR? But yeah, I recognize that there need to be limits. I haven't really thought it through yet, heh.

EDR stands for Endpoint Detection and Response or Remediation.

Think modern antivirus. Less periodic scans and more continuous monitoring for anomalies.