Please also consider including SeaweedFS in the testing.

I wouldn’t use rook if you solely want S3. It is a massively complex system which you really need to invest in understanding or else your cluster will croak at some point and you will have no idea on how to fix it.

IS there a better solution for self-healing S3 storage that you could recommend? I'm also curious what will make a rook cluster croak after some time and what kind of maintenance is required in your experience.

Not used it yet, but RustFS sounds like it has self healing

https://docs.rustfs.com/troubleshooting/healing.html

ceph?

Rook is ceph.

I have unfortunately got a ceph cluster in a bad enough state that I just had to delete the pools and start from scratch. It was due to improper sequencing when removing OSDs, but that is kindof the point is you have to know what you are doing to know how to do things safely. For the most part I have so far learned by blundering things and learning hard lessons. Ceph clusters when mistreated can get into death spirals that only an experienced practitioner can advert through very carefully modifying cluster state through things like upmaps. You also need to make sure you understand your failure domains and how to spread mons and osds across the domains to properly handle failure. Lots of people don’t think about this and then one day a rack goes poof and you didn’t replicate your data across racks and you have data loss. Same thing with mons, you should be deploying mons across at least 3 failure domains (ideally 3 different datacenters) to maintain quorum during an outage.

They explicitly say that top performance is not a goal: «high performances constrain a lot the design and the infrastructure; we seek performances through minimalism only» (https://garagehq.deuxfleurs.fr/documentation/design/goals/)

But it might be interesting to see where the time is spent. I suspect they may be doing fewer things in parallel than MinIO, but maybe it's something entirely different.

>I get the impression that Garage isn’t necessarily focussed on this kind of use case.

I wouldn't be surprised if this will be fixed sometime in the future.

> We can see we're going to have to move away from MinIO in the foreseeable future.

My favorite thing about all of this is that I had just invested a ton of time in understanding MinIO and its Kubernetes operator and got everything into a state that I felt good about. I was nearly ready to deploy it to production when the announcement was released that they would not be supporting it.

I’m somewhat surprised that no one is forking it (or I haven’t heard about any organizations of consequence stepping up anyway) instead of all of these projects to rebuild it from scratch.

You can also just pay for MinIO and treat it like any other commercial dependency, with support and clearer expectations around licensing and roadmap, but forks are a different story: unless there’s a well-funded company or solid consortium behind them, you’re mostly just trading one risk for another.

> with support and clearer expectations around licensing and roadmap

nothing prevents them from hiking pricing, so expectations are not clear.

Somewhat unrelated, but I just looked at the RustFS docs intro[1] after seeing it here. It has this statement:

> RustFS is a high-performance, distributed object storage software developed using Rust, the world's most popular memory-safe language.

I’m actually something of a Rust booster, and have used it professionally more than once (including working on a primarily Rust codebase for a while). But it’s hard to take a project’s docs seriously when it describes Rust as “the world’s most popular memory-safe language”. Java, JavaScript, Python, even C# - these all blow it out of the water in popularity and are unambiguously memory safe. I’ve had a lot more segfaults in Rust dependencies than I have in Java dependencies (though both are minuscule in comparison to e.g. C++ dependencies).

[1]: https://docs.rustfs.com/installation/

I feel dumbfounded. All I've ever heard from rust users, is the equivalent of football fans running up, waving pendants in my face and screaming. So much so, that everything else said seems like the wild fantasies of "our team gonna win".

Then things like this appear:

https://www.phoronix.com/news/First-Linux-Rust-CVE

And I'm all warm and feeling schadenfreude.

To hear "yes, it's safer" and yet not "everyone on the planet not using rust is a moron!!!", is a nice change.

Frankly, the whole cargo side of rust has the same issues that node has, and that's silly beyond comprehension. Memory safe is almost a non-concern, compared to installing random, unvetted stuff. Cargo vet seems barely helpful here.

I'd want any language caring about security and code safety, to have a human audit every single diff, on every single package, and host those specific crates on locked down servers.

No, I don't care about "but that will slow down development and change!". Security needs to be first and front.

And until the Rust community addresses this, and its requirement for 234234 packages, it's a toy.

And yes, it can be done. And no, it doesn't require money. Debian's been doing just this very thing for decades, on a far, far, far larger scale. Debian developers gatekeep. They package. They test and take bug reports on specific packages. This is a solved problem.

Caring about 'memory safe!' is grand, but ignoring the rest of the ecosystem is absurd.

Debian has been doing this for decades, yes, but it is largely a volunteer effort, and it's become a meme how slow Debian is to release things.

I've long desired this approach (backporting security fixes) to be commercialized instead of the always-up-to-date-even-if-incompatible push, and on top of Red Hat, Suse, Canonical (with LTS), nobody has been doing it for product teams until recently (Chainguard seems to be doing this).

But, if you ignore speed, you also fail: others will build less secure products and conquer the market, and your product has no future.

The real engineering trick is to be fast and build new things, which is why we need supply chain commoditized stewards (for a fee) that will solve this problem for you and others at scale!

> Debian has been doing this for decades, yes, but it is largely a volunteer effort, and it's become a meme how slow Debian is to release things.

which is a bit silly considering that if you want fast, most packages land in testing/unstable pretty quickly.

But then you as a consumer/user of Debian packages need to stay on top of things when they change in backwards-incompatible ways.

I believe the sweet spot is Debian-like stable as the base platform to build on top of, and then commercial-support in a similar way for any dependencies you must have more recent versions on top.

> But then you as a consumer/user of Debian packages need to stay on top of things when they change in backwards-incompatible ways.

If you need latest packages, you have to do it anyway.

> I believe the sweet spot is Debian-like stable as the base platform to build on top of, and then commercial-support in a similar way for any dependencies you must have more recent versions on top.

That if the company can build packages properly. Also too old OS deps sometimes do throw wrench in the works.

Tho frankly "latest Debian Testing" have far smaller chance breaking something than "latest piece of software that couldn't figure out how to upstream to Debian"

The difference is between staying on stable and cherry-picking the latest for what you really do need, and being on everything latest.

The latter has a huge maintenance burden, the former is the, as I said already, sweet spot. (And let's not talk about combining stable/testing, any machine I tried that on got into an non-upgradeable mess quickly)

I am not saying it is easy, which is exactly why I think it should be a commercial service that you pay for for it to actually survive.

  > supply chain commoditized stewards (for a fee)

I agree with this, but the open source licenses allow anyone who purchases a stewarded implementation to distribute it freely.

I would love to see a software distribution model in which we could pay for vetted libraries, from bodies that we trust, which would become FOSS after a time period - even a month would be fine.

There are flaws in my argument, but it is a safer option than the current normal practices.

When it is tailored to one customer, that dependency being maintained for you is probably a very particular version you care about. So while copylefted code you can always reshare, it's the timeliness and binary package archives that are where the value really is.

Not dismissing your point, but Looking at the article, it looks like it's in rust unsafe code. Which seems to me to be a point that the rest of the rust code is fine but the place where they turned off the static safety the language provides they got bit.

Hey! Can't I just enjoy my schadenfreude in peace?

I guess the takeaway is that, doubly so, trusting rust code to be memory safe, simply because it is rust isn't sensible. All its protections can simple be invalidated, and an end user would never know.

Um I doubt Debian maintainers look at every single line of code in the packages they maintain.

One might even call the rust community a “cargo cult”

It's hard to take a project seriously if it focuses so much on the language it's written in. As a user, I don't care. Show me the results (bug tracker with low rate of issues), that's what I care about. Whether you program in Rust or C or Java or assembly or PHP.

As a potential user of an open source project, I care a fair bit what language it is implemented in. As an open source project, I preffer projects in languages and ecosystems I am familair and comfortable with. I may need to fix bugs, add features, or otherwise make contributions back to the project, and thus I am more likely to pick a solution in a language I am comfortable with than in a language I am not as comfortable with, given my other needs and priorities are met.

The docs of it and the marketing is a bit of a mess, tho I'm just gonna blame that on culture barrier as the devs are chinese

I agree, although I’m guessing they’re measuring “most popular” as in “most beloved” and not as in “most used.” That’s the metric that StackOverflow puts out each year.

>Secure: RustFS is written in Rust, a memory-safe language, so it is 100% secure

[0]

qed

[0] https://docs.rustfs.com/installation/

Oh wow, they really wrote that. I love Rust but this is clown shit.

I agree that it is a bad idea to describe rust this way but they likely meant memory safety as used in https://www.ralfj.de/blog/2025/07/24/memory-safety.html . Meaning that shared mutable is thread unsafe, I am unsure about Java and JavaScript but I think that almost every language on the popular memory safe list fails this test.

Again the statement is probably still untrue and bad marketing, but I suspect this kind of reasoning was behind it

Of course Rust technically fails too since `unsafe` is a language feature

I don't have an issue with `unsafe` - Java has the mythical unsafe object, C# has it's own unsafe keyword, Python has ffi, etc. The title of that blog post - that there is no memory safety without thread safety - is not quite true and it acknowledges how Java, C#, and Go have strong memory safety while not forbidding races. Even the "break the language" framing seems like special pleading; I'd argue that Java permitting reading back a sheared long (64-bit) integer due to a data race does not break the language the same way writing to a totally unintended memory area or smashing the stack does, and that this distinction is useful. Java data races that cause actual exploitable vulnerabilities are very, very rare.

I am Elvin, from the RustFS team in the U.S.

Thanks for the reality check on our documentation. We realize that some of our phrasing sounded more like marketing hype than a technical spec. That wasn’t our intent, and we are currently refining our docs to be more precise and transparent.

A few points to clarify where we’re coming from: 1. The Technical Bet on Rust: Rust wasn’t a buzzword choice for us. We started this project two years ago with the belief that the concurrency and performance demands of modern storage—especially for AI-driven workloads—benefit from a foundation with predictable memory behavior, zero-cost abstractions, and no garbage collector. These properties matter when you care about determinism and tail latency. 2. Language Safety vs. System Design: We’re under no illusion that using a memory-safe language automatically makes a system “100% secure.” Rust gives us strong safety primitives, but the harder problems are still in distributed systems design, failure handling, and correctness under load. That’s where most of our engineering effort is focused. 3. Giving Back to the Ecosystem: We’re committed to the ecosystem we build on. RustFS is a sponsor of the Rust Foundation, and as we move toward a global, Apache 2.0 open-source model, we intend to contribute back in more concrete ways over time.

We know there’s still work to do on the polish side, and we genuinely appreciate the feedback. If you have specific questions about our implementation details or the S3 compatibility layer, I’m happy to dive into the technical details.

Those rates are peanuts considering that a decade ago saturating 40G, per core, was more than reasonable via standard userspace networking, with atleast a few copies in the datapath.

passing blocks of memory around vs referencing filesystem/database, ACLs, authentication and SSL

Having just finished a "hobby size" setup of Rook-Ceph on 3 n100 mini pcs, with every service to fit in a couple hundred MB of ram (one service needs up to 3Gb when starting, but then runs around 250MB) I'd ask why not ceph?

At work I'm typically a consumer of such services from large cloud providers. I read in few places how "difficult" it is, how you need "4GB minimum RAM for most services" and how "friends do not let friends run Ceph below 10Gb".

But this setup runs on a non dedicated 2.5Gb interface (there is VLAN segmentation and careful QoSing).

My benchmarks show I'm primarily network latency and bandwidth limited. By the very definition you can't get better than that.

There were many factors why I chose Ceph and not Garage, Seaweed or MinIo. (One of the biggest is that ceph does 2 birds with one stone for me - block and object).

Ceph is far higher on RAM usage and complexity. Yeah if you need block storage in addition it's a good choice, but for anything smaller than half a rack of devices it's kinda overkill

Also from our experience the docs outright lie about ceph's OSD memory usage and we've seen double or more than what docs claim (8-10GB instead of 4)

My small adventure with rustfs is that it is somewhat underbaked at the moment.

And also it is already rigged for a rug-pull

https://github.com/rustfs/rustfs/blob/main/rustfs/src/licens...

yeah, their docs look pretty comprehensive, but there's a disturbing number of 404s that scream "not ready for prime-time" to me.

from https://rustfs.com/ if you click Documentation, it takes you to their main docs site. there's a nav header at the top, if you click Docs there...it 404s.

"Single Node Multiple Disk Installation" is a 404. ditto "Terminology Explanation". and "Troubleshooting > Node Failure". and "RustFS Performance Comparison".

on the 404 page, there's a "take me home" button...which also leads to a 404.

Thanks for flagging this and for taking the time to point out the broken links. We open-sourced RustFS only a few months ago, and while we’ve been heavily focused on getting the core system to GA, that has admittedly created some documentation debt.

We’re actively reviewing the docs and cleaning up any 404s or navigation issues we can find. For the specific 404 you mentioned, we haven’t been able to reproduce it on our end so far, but we’re continuing to investigate in case it’s environment- or cache-related.

On the licensing side, we want to be clear that we’re fully committed to Apache 2.0 for the long term.