Curious -- any reason you didn't just go with a single machine export + expansion disk shelves on something like ZFS? Installing a MinIO gateway would also act as a bare drop-in for S3 too.

Asking since we're in the same position as yourself w/ high double-digit disks trying to figure out our plan moving forward. Right now we're just using a very large beefy node w/ shelves. ZFS (via TrueNAS) does give us pretty good guarantees on failed disks + automated notifications when stuff goes wrong.

Obviously a single system won't scale past a few hundred disks so we are looking at alternatives including Ceph, GlusterFS, and BeeGFS. From the outside looking in, Ceph seems like it might be more complexity than it's worth until you hit the 10s of PB range with completely standardized hardware?

Some of our rendering processes take multiple days to complete, and the blackbox software we use doesn't have a pause button. So it's not that we're in need of 99.99999% uptime, but there's actually never a moment where rebooting a machine would be convenient (or indeed cost us money). Being distributed over nodes means I can reboot them and the processes are not disrupted.

for k8s there is also kadalu btw. which is based on glusterfs, but simplified.

HDFS doesn't really work as a normal filesystem. I think some other commenters pointed out the challenges with FUSE.

If I recall correctly there isn't really a way to modify an existing file via HDFS, so you'd have to copy/edit/replace. Append used to be an issue, but that got sorted out a few years back.

Erasure coding is available in the latest versions. Which helps with replication costs.

I think, HDFS may just be a simpler setup than other solutions. (which is to say its not all that simple, but easier than some other choices). And I wouldn't use HDFS as a replacement for block storage, which is something I've seen done with Ceph.

Thanks, we actually use Ceph as a straight up filesystem, that gets mounted on linux machines and then exposed to our windows based processing nodes (they are human operated) over SMB. I think that explains why HDFS is not a good fit for us.

What about S3 didn't meet your use case? I don't work for AWS. I don't care if they lose business, I am interested in how different companies parse their requirements into manage vs. rent.

One aspect is that we have a lot of data that has PII in it, and we feel safer if we anonymise that locally before sending it into the cloud. Once the data is cleaned up it's actually sent to GCS for consumption in our product). Another aspect is that this data has to be accessible as windows fileshares (i.e. SMB) to our data processing team. The datasets are in the range of several 100's of GB to several TB, each of the team members works on several of those datasets per day. This would strain our uplink too and maybe the bandwidth would be costly as well.

If you are writing a ton of small files (we have billions of audit blobs we write) the API put costs can quickly creep up on your. We pay much more for those than on the actual storage costs. If you want to use tags on your objects, they charge you per tag per object per month - again, another huge cost. We missed that when pricing S3 out, and needed to do a project to pull out all of the tags we had, and are currently working on batching up multiple blobs into one larger blobs to hopefully reduce our API costs by an order of magnitude. This is purely a cost decision for us, adding complexity to our application and its operation. S3 seems better suited for fewer larger files. Our backups and other use cases like that work perfectly.

HDFS has pretty much all of Ceph's flaws plus it has a non-scalable metadata server, the "NameNode". If you're already up and running with Ceph I can think of no reason to abuse yourself with HDFS.

We're spinning up a medium sized Proxmox clusters (~50 nodes in total) to replace our aging Xen clusters. I saw Ceph is available on the Proxmox platform, but was hesitant to make all the VM storage backed by Ceph (throwing all the eggs into a single basket).

What were some of the other hurdles you faced in your Ceph deployment?

We've been playing around with migrating our bare metals to proxmox as well. Though one main argument, being able to reboot/manage crashed GPU accelerated nodes, was invalidated by proxmox (KVM?) itself crashing whenever the GPU would crash, so it didn't solve our core problem. This is of course also due to that we're not using industrial components, but it is what it is.

I found Ceph's error messages very hard to debug. Just google around a bit for the manuals of how to deal with buggy or fully defective drives. There's a lot of SSH'ing in, running vague commands looking up id's of drives and matching them to linux device mount points and reading vague error logs.

To me as a high level operator it feels it should be simple. If a drive supplies a block of data, and that data fails its checksum, it's gone. The drive already does its very best internally to cope with physical issues, if the drive couldn't come up with valid data, it's toast or as close to toast as anyone should be comfortable with. So it's simple, fail a checksum, out of the cluster, send me an e-mail, I don't get why Ceph has to be so much more complicated than that.

I found proxmox not to be very user friendly growing to such cluster sizes. Proxmox itself has been very stable and supports pretty much anything but the GUI is not that great if you have many nodes and VMS, and the API can be lacking. However, using ceph as a backing store for VM images is pretty easy in proxmox. I have not used the cephFS stuff. I used it in a separate cluster both physically and standalone (not using proxmox integration).

So RBD is easy, S3 is somewhat more complicated as you need to run multiple gateways, but still very doable. The FS stuff also needs extra daemons, but I have not yet tested it.

You have to use something like FUSE to mount HDFS, if that is your intention. It's not really like Ceph. Unless your app is written to use the HDFS API directly it's going to be a bigger rigmarole to store stuff.

Did you not evaluate linstor?

Thanks, I didn't but it looks interesting, I'll research it later.