Re:Longhorn – A Kubernetes-Native Filesystem
by positisop on Wednesday September 10, 2025 @06:40PM
(#45205071)
Longhorn is a poorly implemented distributed storage layer. You are better off with Ceph.
[ Parent
| on HN ]
Re:Longhorn – A Kubernetes-Native Filesystem
by willbeddow on Wednesday September 10, 2025 @07:40PM
(#45205696)
have not used longhorn, but we are currently in the process of migrating off of ceph after an extremely painful relationship with it. Ceph has fundamental design flaws (like the way it handles subtree pinning) that, IMO, make more modern distributed filesystems very useful. SeaweedFS is also cool, and for high performance use cases, weka is expensive but good.
Re:Longhorn – A Kubernetes-Native Filesystem
by __turbobrew__ on Wednesday September 10, 2025 @08:16PM
(#45205995)
Are there any distributed POSIX filesystems which don’t suck? I think part of the issue is that POSIX compliant filesystem just doesn’t scale, and you are just seeing that?
Re:Longhorn – A Kubernetes-Native Filesystem
by huntaub on Thursday September 11, 2025 @12:11AM
(#45207698)
Basically, we are building this at Archil (https://archil.com). The reason these things are generally super expensive is that it’s incredibly hard to build.
Re:Longhorn – A Kubernetes-Native Filesystem
by scheme271 on Wednesday September 10, 2025 @10:33PM
(#45207052)
I think Lustre works fairly well. At the very least, it's used in a lot of HPC centers to handle large filesystems that get hammered by lots of nodes concurrently. It's open source so nominally free although getting a support contract from specialized consulting firm might be pricey.
Re:Longhorn – A Kubernetes-Native Filesystem
by latchkey on Wednesday September 10, 2025 @11:37PM
(#45207498)
https://www.reddit.com/r/AMD_Stock/comments/1nd078i/scaleup_...
You're going to have to open the image and then go to the third image. I thought it was interesting that OCI pegs Lustre at 8Gb/s and their high performance FS at much higher than that... 20-80.
Re:Longhorn – A Kubernetes-Native Filesystem
by scheme271 on Thursday September 11, 2025 @08:14PM
(#45217436)
That's 8Gb/s per TB of storage. The bandwidth is going to scale up as you add OSTs and OSSs. The OCI FS maxes at 80Gb/s per mount target.
Re:Longhorn – A Kubernetes-Native Filesystem
by willbeddow on Wednesday September 10, 2025 @08:34PM
(#45206146)
weka seems to Just Work from our tests so far, even under pretty extreme load with hundreds of mounts on different machines, lots of small files, etc... Unfortunately it's ungodly expensive.
Re:Longhorn – A Kubernetes-Native Filesystem
by q3k on Wednesday September 10, 2025 @07:45PM
(#45205745)
That sounds more like a CephFS issue than a Ceph issue.
(a lot of us distrust distributed 'POSIX-like' filesystems for good reasons)
Re:Longhorn – A Kubernetes-Native Filesystem
by yupyupyups on Wednesday September 10, 2025 @06:46PM
(#45205142)
I've heard Ceph is expensive to run. But maybe that's not true?
Re:Longhorn – A Kubernetes-Native Filesystem
by keeperofdakeys on Wednesday September 10, 2025 @07:15PM
(#45205445)
Ceph overheads aren't that large for a small cluster, but they grow as you add more hosts, drives, and more storage. Probably the main gotcha is that you're (ideally) writing your data three times on different machines, which is going to lead to a large overhead compared with local storage.
Most resource requirements for Ceph assume you're going for a decently sized cluster, not something homelab sized.
Re:Longhorn – A Kubernetes-Native Filesystem
by jauntywundrkind on Wednesday September 10, 2025 @07:06PM
(#45205338)
I'm only just wading in, after years of intent. I don't feel like Ceph is particularly demanding. It does want a decent amount of ram. 1GB each for monitor, manager, and metadata, up to 16GB total for larger clusters, according to docs. But then each disk's OSD defaults to 4gb, which can add up fast!! And some users can use more. 10Gbe is recommended and more is better here but that seems not unique to ceph: syncing storage will want bandwidth. https://docs.ceph.com/en/octopus/start/hardware-recommendati...
Re:Longhorn – A Kubernetes-Native Filesystem
by xyzzy123 on Wednesday September 10, 2025 @07:18PM
(#45205468)
For me it was the ram for the OSDs, 1GB per 1TB but ideally more for SSDs...
Re:Longhorn – A Kubernetes-Native Filesystem
by westurner on Wednesday September 10, 2025 @07:16PM
(#45205455)
This from 2023 says: https://www.redhat.com/en/blog/ceph-cluster-single-machine :
> All you need is a machine, virtual or physical, with two CPU cores, 4GB RAM, and at least two or three disks (plus one disk for the operating system).
Re:Longhorn – A Kubernetes-Native Filesystem
by master_crab on Wednesday September 10, 2025 @10:26PM
(#45207015)
It’s going to do a good job saturating your lan maintaining quorum on the data.