Its possible you'd end up in situations where the OS would need to retain old file contents if there's both a pending read from a subscriber and subsequent writes. But in most cases, this is fine.
Going back to my original comment, this is the sort of reason I'd like to try this in my own OS kernel rather than try and solve all these problems immediately in linux. It'll give me a chance to try out a few different solutions and see if I can find something that works well.
Meaning a process could start loading and reading while queuing writes and only writes would block while another process is locking access to the executable to scan it. If the scan succeeds the writes are committed and if it fails it could revoke execution privileges and all the work is undone.
Lot of hand waving there, but when you're not committed to existing file system semantics there is a lot of room for innovation.
Think modern antivirus. Less periodic scans and more continuous monitoring for anomalies.
Also, how this would be done would probably change depending on whether you're using a spinning-rust HD or an SSD, or a RAID array.
Both Oracle RDBMS and older versions of Microsoft SQL Server had the option to use "raw disk devices" / "raw partitions" instead of the filesystem. It had some obvious justifications such as avoiding the "wasteful double buffering" of the file system cache being redundant to the database's cache and avoiding "unnecessary" extra i/o abstraction layers.
Microsoft later got rid of that option because it wasn't worth the tradeoff of a small performance gain while losing the easier management aspects of NTFS file system.
Abstracting away your hard disk is literally the job of your filesystem.
I agree with you - you could hack around this for a big database by putting it on a dedicated partition (even though you shouldn't need to). But I'm also thinking about applications with little databases that need to live inside the filesystem. They shouldn't need to embed sqlite in order to survive a crash.
A different syscall API would allow you to write a small, fast, efficient database engines (like redis or mysql) without needing all of the complex, performance gobbling tricks that these databases currently have.
A better filesystem API would help databases everywhere.
if you make direct and locking more robust, and allowed an application to turn off the kernel cache, you would only really need to interact with the filesystem after open in order to change your extents.
+1. this is clearly a place where we can have nice things and performance
Having a OS+filesystem running a virtual OS+filesystem running a database has all kinds of redundant caching, and we could systematically skip a layer here.
I can search millions of records in a split second in SQL. But searching records on my hard disk takes enormously more time. The solutions out there for speeding it up (like Windows Indexing service) rely on asynchronous indexing that's patched on top of the filesystem instead of integrated realtime within it, and are subsequently brittle (in my experience), slow down your computer at inconvenient times, and too often aren't instantly up date.
Being able to have stored procedures execute when changes trigger would be another interesting aspect that could take the place of event filters and file watchers (especially if critical portions could be configured to happen sequentially & atomically, akin to IRQ handlers).
I feel like databases have gotten lots of love over the decades and are super optimized (and by this point are pretty dang reliable). Filesystems feel like they lost their time in the spotlight and have floundered with less innovation.
Hardware that calculates CRC's of blocks in realtime would also be awesome (imagine what it would do for sync). The ironic thing is your HDD already does this, there's just no way to tap into that information at the software / filesystem driver level.
Microsoft made an attempt back in 2003 at creating a "smarter" file system with rich metadata based on a real database engine (MS SQL Server) ... but eventually abandoned the project: https://en.wikipedia.org/wiki/WinFS
WinFS wasn't necessarily going to completely replace NTFS but Microsoft did have ambitious plans for its integration into everything.
It's interesting that other platforms like Unix/Linux/Apple that were designing "next generation" file systems like ZFS/ext4/APFS after the failed WinFS project had a chance to try adding their own WinFS smart metadata features to but they didn't.
The later fragmentation of user data being spread around in the cloud such as AWS objects, Apple iCloud, Backblaze backups, etc instead of every file saved on home computers makes a "universal rich metadata index stored in a database engine" to be a cross-platform standard by all operating systems less realistic today than back in 2003. Therefore, the core os file systems remain a "dumb blob of bytes" with a separate layer of metadata index db in vendor-specific schemes like Apple Spotlight db, Windows Search in ".edb" file, etc.
All I want is dumb files with something like SQLite replacing or augmenting the allocation table. Then sprinkle on some new capabilities incrementally.
The 'file table' contains a 64 byte record for each object and has some very unique characteristics that make searches incredibly fast (e.g. it can find all the photos in under a second, even if there are 20 million of them among 200 million files). You can also attach various metadata 'tags' to each object and find matching objects based on them.
The project is in open beta and anyone can download it and try it out on their own computer.
It has relational database-like semantics for metadata.
You probably won't supplant Linux with your toy OS for your own research, but for a software engineer, writing a OS is the closest thing we have to the exhilarating feeling of seeing your code affect real hardware. You haven't lived until a bug of your own causes a triple-fault exception that causes a literal system shutdown.
I started this career writing my own tiny OS in my teens (https://github.com/1player/klesh) and for 20 years I've been wanting to go back, with all the experience I have accumulated.
Just don't make it another UNIX, please. We have enough of those already ;)
Hah thanks for the encouragement! Don't worry, there's no danger on that front. I'm pretty thoroughly bored of the "everything is a file" motif.
I do like unix's idea of making large programs out of small programs that can be composed together. But unix's raw byte streams make a terrible messaging bus.
Which also means inventing a language to build this novel paradigm. As Dan Ingalls said, an operating system is everything that doesn't fit in a language.
I am looking forward to finding the time for this long and quixotic quest :-)
I present Crazierl: https://crazierl.org/
My crazy hobby OS vision of boot to Erlang. I've been working on this for a couple years now, recently made it public as part of a PR to v86 [1] (which is amazing, by the way). 32-bit x86, has drivers for virtio and rtl-8168 (but not ne2k, so no networking in v86, sadly), works with SMP. Probably not better than running beam as init with a FreeBSD kernel, but was more fun?
On the OS I work on, I deliberately added a triple fault trigger as a fallback method of rebooting the system if the "normal" methods fail, which I have seen before.
To anyone looking to play with writing their own OS, definitely just do it. It's a lot of fun, and there's almost no limit to how much you can learn. With QEMU and bochs, and VirtualBox, etc... it's not that hard getting started. If you want to play around without starting from scratch, but find Linux source too complicated, give OpenBSD a look. Not that Linux is that hard, but it takes some getting used to how they do things. I find OpenBSD considerably more approachable.
Unless you want to muck about with toy robotics or home automation.
I'd love to try it out when (if ever) I have the time.
kqueue is very limited in terms of what you can subscribe to, and things like filesystem watching usually requires subsequent filesystem calls to see whats actually changed. And that in turn introduces race conditions if the data is changed again before your fs syscalls make it through.
I think there are cleaner APIs out there. I'd like to see if I can find one.
If you have all of your programs reside in the same virtual memory space (which is the implication with running them in ring 0), then you get a unique performance advantage in that you don't have to invalidate the TLB. I have no idea how much the amortized performance impact of virtual memory is, unfortunately, so the gains might be trivial, but who knows.
I suspect that programs which make a lot of kernel calls would run faster in the kernel in wasm, and programs that are more compute / memory bound will run faster as native processes. It'd be fun to play around and find where the tradeoffs lie for different applications.
I'm not sure if I understood you correctly.
But if programs were written in wasm, that problem goes away. Wasm prevents bad memory accesses by doing memory protection in software. (Every array or pointer access is bounds checked).
Fuchsia's "Signals" are designed in that direction. You get a "handle" to a resource, and the resource has a 32-bit integer with signal bits that you could subscribe to. The meaning of each bit is different for each type of resource.
(In general operating system terminology, "Signal" means just a message without any payload that would have required any memory allocation. There are many ways to do it, not just the Unix way. Message ports and other abstractions are sometimes built on top of types of signals)
Do you have a link for this? It's the first time I'm seeing signals described in terms of lack of memory allocation. I always thought the word "signal" basically meant user-mode interrupts - asynchronously hijacking a thread.
I strongly encourage you to do this!
I think writing your own OS is one of the things every dev should do at least once, just for the educational value. The others are writing a game and writing a compiler and linker.
Go one step further and have the kernel expose a much more structured view of storage to user space by default. 99.9% of user space programs I've written don't want to deal with file IO, they want a database, or some other way for arbitrary in memory objects to be persisted and shared between processes. There's so much room to improve on the 1970s era IO primitives we use by default.
'file' objects were effectively database tables, and even things like source code for programs were stored as such.
Eventually IBM added the IFS (integrated file system) to provide a more traditional POSIX-adjacent filesystem that was in fact just a bunch of bits, and it feels weird and out of place with the rest of the system as a result.
There was at least one Unix startup (IIRC it was Tolerant Systems) in mid 80s that applied transactional semantics to filesystem operations. I did some contract work for them and found the performance was absolutely abysmal and the system had some deadlocks. Now may be their implementation was not great but I suspect the converse is also true: database semantics are terrible for filesystems :-)
To be frank, judging by the code I’ve seen over the years, most engineers don’t have a lot of experience or skill at making software run fast.
1. Have either database-like transactions or (at a minimum) write barriers for file operations. The current filesystem semantics are terrible for databases and other applications which need to not corrupt their data after a crash. fsync() is too heavy, because you can't issue subsequent writes while waiting for fsync. Write barriers have better ergonomics (so, less complex, less buggy database implementations). And they're finer grained - so databases can also run much faster. Seriously, I could rant for hours about this. It drives me nuts.
2. Use event queue like semantics for subscriptions. There's dozens of OS APIs in which the OS exposes some data to applications where the data changes over time. Eg, filesystem watching, USB device insertion/removal, Bluetooth, watching a network device, or a socket, or APIs for htop to see the set of active processes. Each of these APIs has an entirely bespoke - and oftentimes idiosyncratic - API to get changes to that data over time. Sometimes you need to poll and parse a file in procfs. Sometimes there's syscall based APIs. And some of the subscription APIs are just broken sometimes, and they'll fail to notify you about changes sometimes.
I want to try replacing all of that with a simple, unified API which lets you say "tell me the state of <kernel resource X> and tell me when it changes". The kernel then replies with the object's current state and I get messages notifying my application about changes. We should be able to use the same API for every type of kernel object.
It would also be interesting to try making all userspace programs be wasm bundles and run them in ring0. But I suspect that will run slower on modern hardware compared to using native binaries and context switching. ... Maybe. It might depend on the application.
I think this stuff could be done in linux. Event queue style subscriptions would work especially well with io_uring. But I'm kind of excited by the idea of just noodling myself and seeing where I end up. Making stuff is fun.