building fs with such large features set is just untrivial task, and btrfs one of very few who made it, so it is absolute success story.
In what sense?
The reason I ask is because I'm trying to tease out if you have architectural problems with the way the filesystem is designed, or if you simply think it's unreliable.
an internet community project to write an entire operating system from scratch using some newfangled programming language is literally the final boss of yak shaving. there is no reason to do it other than "it's fun" and of course writing a filesystem for it would be fun.
For me, the things that would make it just perfect would be more ergonomic Cap'n Proto support (eliminate a ton of fiddly code for on disk data structures), and dependent types.
i suspect the linux stuff would be far more space and time efficient, but we won't know until projects like this mature more.
Now, the engineering effort required to rewrite or redo in Rust, that's a different story of course.
It's _so_ _much_ _better_. I can use async, maps, iterators, typesafe deserialization, and so on. All while not using any dynamic allocations.
With full support from Cargo for repeatable builds. It's night and day compared to the regular Arduino landscape of random libraries that are written in bad pseudo-object-oriented C++.
Calling it "optimized" is a stretch. A veeeery big one. The low-level code in some paths is highly optimized, but the overall kernel architecture still bears the scars of C.
The most popular data structure in the kernel land is linked list. AKA the most inefficient structure for the modern CPUs. It's so popular because it's the only data structure that is easy to use in C.
The most egregious example is the very core of Linux: the page struct. The kernel operates on the level of individual pages. And this is a problem in case you need _a_ _lot_ of pages.
For example, when you hibernate the machine, the hibernation code just has a loop that keeps grabbing swap-backed pages one by one and writing the memory to them. There is no easy way to ask: "give me a list of contiguous free page blocks". Mostly because these kinds of APIs are just awkward to express in C, so developers didn't bother.
There is a huge ongoing project to fix it (folios). It's been going for 5 years and counting.
Is this reasoning really true? A quick search reveals the availability of higher-level data structures like trees, flexible arrays, hashtables, and the like, so it's not as if the linux kernel is lacking in data structures.
Linked lists have a few other advantages - simplicity and reference stability come to mind, but they might have other properties that makes them useful for kernel development beyond how easy they are to create.
Well, yes. The kernel _now_ has all kinds of data structures, but you can look at the documentation from 2006 and see the horror of even the simplest rbtrees back then: https://lwn.net/Articles/184495/
A simple generic hashtable was added only in 2013!
> Linked lists have a few other advantages - simplicity and reference stability come to mind, but they might have other properties that makes them useful for kernel development beyond how easy they are to create.
The main property is that it's easy to use from C. Think about growable vectors as an exercise.
(XArray in regular C-based Linux also kinda qualifies)
I don't understand that statement. Linked lists are no easier or harder to use than other data structures. In all cases you have to implement it once and use it anywhere you want?
Maybe you meant that linked lists are the only structure that can be implemented entirely in macros, as the kernel likes to do? But even that wouldn't be true.
You can iterate through it fine, it's just an array after all. But then you want to add an element to it. At this point you need to handle reallocation. So you need to copy or move the elements from the old array. But this will break if the elements are not simple data structures.
So you need to have a copy function. There are copy constructors in C++, but not in C. So you need to reinvent them.
We see this in the compiler frameworks that predated the C++ standard, and whose ideas lived on Java and .NET afterwards.
Too many C++ libraries are written for a C++ audience, when they could be just as ergonomic as in other high level languages, and being C++, there could be a two level approach, but unfortunately that isn't how it rolls.
There needs to be projects like that for any kind of innovation to happen.
Why leave at that? One day it can be production ready.
The filesystem is so deeply connected to the OS I bet there's a lot of horror around swapping those interfaces. On the contrary, I've never heard anything bad about DragonflyBSD's HAMMER. But it's basically assumed you're using DragonFlyBSD.
Would I keep a company's database on a new filesystem? No, nobody would know how to recover it from failed disk hardware.
This isn't really my area but a Rust OS using a ZFS-like filesystem seems like a lot of classic Linux maintainer triggers. What a funny little project this is. It's the first I've heard of Redox.
Edit: reminds me of The Tarpit chapter from the Mythical Man Month
> The fiercer the struggle, the more entangling the tar, and no beast is so strong or so skillful but that he ultimately sinks.
Filesystems, as complex as they are, aren't full of traps like encryption is. Still plenty of subtle traps, don't get me wrong: you have to be prepared for all kinds of edge cases like the power failing at exactly the wrong moment, hardware going flaky and yet you have to somehow retrieve the data since it's probably the only copy of someone's TPS report, that sort of thing. But at least you don't have millions of highly-motivated people deliberately trying to break your filesystem, the way you would if you rolled your own encryption.
The only connection is that writing custom encryption is a thing that smart people like to try their hand at, but its success is defined by the long tail of failure cases not by the cleverness of the happy path. I agree 100% with what rmunn said.
As I said I'm not a filesystem person, but my sense is that filesystem difficulty is also dominated by the long tail of failure cases and for similar reasons. Failure in encryption means you lose control of your data, failure in filesystems mean you lose your data (or maybe you lose liveness/performance) [0]
But really I just meant it in the sense that it's a journey people often go down underestimating just how long it takes. So it's a sort of trap from the project management perspective.
> I'm confused about what 'rolling your own encryption' means at an abstraction level
It cuts through many abstractions. You should definitely not define your own crypto primitives. You also shouldn't define your own login flow. You shouldn't design a custom JWT system, etc. You probably shouldn't write your own crypto library unless there's not one in your language, in which case you should probably be wrapping a trusted library in C or C++, etc. The higher you go in abstraction, the more it's okay to design an alternative. But any abstraction can introduce a weakness so the risk is always there.
[0] Ordinarily you still have backups, which makes file system failures potentially less final than encryption failures. But what if the filesystem holding your backup root keys fails. Then the encryption wasn't a failure but you've potentially crypto shredded your entire infrastructure.
What choice of paint you throw into the tarpit makes zero difference.
If you're going to have to write it from scratch (since rust), might as well make your own.
Worst case this doesn't work. Best case, this works amazingly well. I think there's some valid reason for optimism here give other hard things that Rust has been used for in the past few years.
The people who just think OSes would be better with more Rust in them, but aren't looking to reinvent from first principles are in the Rust for Linux universe.
And you know what, that's fine. Linux started out as a hobby project with similar origins before it became a big serious OS.
If your goal is to gain wide adoption fast, that is a bad idea.
Yeah, to clarify, that was how I took it. I'm not concerned about Redox OS failing to get adoption. And I love that they're exploring new things and following their interests.
I was thinking more as someone with ADHD and widespread interests. It feels a bit like saying "I'm learning the oboe. But first I want to learn how to make an oboe." Both are cool goals, but if you do the second you may never get to the first in a meaningful way.
That may be okay, I was just intending to express surprise at a detour that could last a significant fraction of the development teams' lives. Especially since as the default file system, delays or breakages in the filesystem can delay or break the OS project as a whole.
https://gitlab.redox-os.org/redox-os/redoxfs/-/commit/bc2d90...
Rust people love doing that, let them. Let them introduce logic bugs across the codebase. Sounds fun.
* inotify implementation is insufficiently atomic, so event subscribers can fail to receive notifications under certain conditions.
* License/patent encumbrance prevents modern operating systems from implementing/distributing a common next-gen FS.
* ZFS has native encryption in theory but it was bolted on later and has numerous buggy interactions the powerful zfs send/recv.
* ZFS native encryption is architecturally incapable of protecting metadata like filenames.
There is lots of room for innovation, and with modern tools it shouldn't take decades to build a production-ready driver for any one ISA (presumable x86_64.) If this project wants to pilot a superior option, I'm all for it!
* RedoxFS is MIT licensed
* RedoxFS supports all features with or without full disk encryption
* RedoxFS encryption includes all metadata
Isn't writing a robust file system something that routinely takes on the order of decades? E.g. reiserfs, bcachefs, btrfs.
Not to rain on anyone's parade. The project looks cool. But if you're writing an OS, embarking on a custom ZFS-inspired file system seems like the ultimate yak shaving expedition.