Why is it a bad idea?

Traditional storage has one RAID Controller with a lot of disks hanging off it, traditional storage servers have one server fronting to the network with a lot of disks inside it, traditional SANs have one head with a couple of controllers and a lot of disk shelves hanging off the back, this design bottlenecks at the single front end.

Ceph is designed not to have a single front end, any node in the cluster can take data in and relay it to the target node where the data is stored. To avoid having a single point of failure where all the nodes look to one lookup table to say where the data lives, they all use the algorithm to choose where the data lives. This is guided by admin rules known as the CRUSH Map where to say things like:

- these storage servers are grouped together by some label (e.g. into a rack, using the same power feed, in a data centre, same site)

- I want N copies of data, and separated over different redundancy / failure boundaries like different racks or different sites.

The result of this design is a massively parallel system that can ingest a lot of data in parallel streaming using every node’s network bandwidth, but quite a lot of internal bandwidth shuffling data around, and quite slow in latency because there’s a layer of calculation and mapping and inter-node network traffic for every block read/written.

The result of this design is that the resting place for a piece of data is deterministic, like a hash of the data guided by placement rules, so that every node follows the algorithm and arrives at the same resulting location where the data needs to go/come from and there is no single controller which can remap the storage “ohh this location is unbalanced and filling up, put it there instead“. As a result of that, the cluster can’t really run with different size disks very well or very easily, and it’s “as full as the fullest disk” - one full disk will put the entire cluster into read-only mode until you fix it. So it runs best with raw storage around half full or less, and keep space free to move data in case of failure - if you said store data per rack, a whole rack’s worth of rebuilding might be needed.

It’s got a design where the units of storage are OSDs which correspond roughly to disks/partitions, each one has a daemon controlling it, on top of that Ceph pulled the storage together as RADOS - reliable autonomic distributed object store - and on top of that you can layer user-visible storage such as CephFS filesystem, Amazon S3 compatible object storage, or a layer that presents as a block device which can be formatted with XFS/etc. so there’s another layer there which doesn’t help latency.

And it needs monitor daemons and management daemons overseeing the whole cluster to keep track of failed storage units and make that available to all nodes, and management/monitoring services.

So overall it has a high administrative complexity to setup, and needs a lot of hardware/storage to get the best out of it. Any small use case will be better off with a traditional SAN or Linux JBOD (just a box of disks) server, easier to manage and higher performance, lower latency, and then move to two file servers for network parallelism or sync between them.

Ceph can run with HDDs but has a separate place for metadata to be on SSD but if that’s not sized well then it spills over to HDD and performance worsens. I inherited some Ceph management work and it’s gone into my list of things I respect from a distance but don’t enjoy. Sad because it’s a very interesting system, but there was no management appetite for investing in it so it was overloaded, underperforming, and caused many problems.

It can be all set up on three nodes I think, so you can play with it, but I wouldn’t think that’sa good production design. It needs a lot of nodes and some high effort setup to really spread itself out and you need a use case that benefits from petabytes of storage and streaming writes in parallel - there are some talks on YouTube by people who managed and upgraded it at CERN as targets of particle accelerators data which are quite interesting.

While most of what you speak of re Ceph is correct, I want to strongly disagree with your view of not filling up Ceph above 66%. It really depends on implementation details. If you have 10 nodes, yeah then maybe that's a good rule of thumb. But if you're running 100 or 1000 nodes, there's no reason to waste so much raw capacity.

With upmap and balancer it is very easy to run a Ceph cluster where every single node/disk is within 1-1.5% of the average raw utilization of the cluster. Yes, you need room for failures, but on a large cluster it doesn't require much.

80% is definitely achievable, 85% should be as well on larger clusters.

Also re scale, depending on how small we're talking of course, but I'd rather have a small Ceph cluster with 5-10 tiny nodes than a single Linux server with LVM if I care about uptime. It makes scheduled maintenances much easier, also a disk failure on a regular server means RAID group (or ZFS/btrfs?) rebuild. With Ceph, even at fairly modest scale you can have very fast recovery times.

Source, I've been running production workloads on Ceph at fortune-50 companies for more than a decade, and yes I'm biased towards Ceph.

I defer to your experience and agree that it really depends on implementation details (and design). I've only worked on a couple of Ceph clusters built by someone else who left, around 1-2PB, 100-150 OSDs, <25 hosts, and not all the same disks in them. They started falling over because some OSDs filled up, and I had to quickly learn about upmap and rebalancing. I don't remember how full they were, but numbers around 75-85% were involved so I'm getting nervous around 75% from my experiences. We suddenly commit 20TB of backup data and that's a 2% swing. It was a regular pain in the neck, stress point, and creaking, amateurishly managed, under-invested Ceph cluster problems caused several outages and some data corruption. Just having some more free space slack in it would have spared us.[1]

That whole situation is probably easier the bigger the cluster gets; any system with three "units" that has to tolerate one failing can only have 66% usable. With a hundred "units" then 99% are usable. Too much free space is only wasting money, too full is a service down disaster, for that reason I would prefer to err towards the side of too much free rather than too little.

Other than Ceph I've only worked on systems where one disk failure needs one hotspare disk to rebuild, anything else is handled by a separate backup and DR plan. With Ceph, depending on the design it might need free space to handle a host or rack failure, and that's pretty new to me and also leads me to prefer more free space rather than less. With a hundred "units" of storage grouped into 5 failure domains then only 80% is usable, again probably better with scale and experienced design.

If I had 10,000 nodes I'd rather 10,100 nodes and better sleep than playing "how close to full can I get this thing" and constantly on edge waiting for a problem which takes down a 10,000 node cluster and all the things that needed such a big cluster. I'm probably taking some advice from Reddit threads talking about 3-node Ceph/Proxmox setups which say 66% and YouTube videos talking about Ceph at CERN - in those I think their use case is a bursty massive dump of particle accelerator data to ingest, followed by a quieter period of read-heavy analysis and reporting, so they need to keep enough free space for large swings. My company's use case was more backup data churn, lower peaks, less tidal, quite predictable, and we did run much fuller than 66%. We're now down below 50% used as we migrate away, and they're much more stable.

[1] it didn't help that we had nobody familiar with Ceph once the builder had left, and these had been running a long time and partially upgraded through different versions, and had one-of-everything; some S3 storage, some CephFS, some RBDs with XFS to use block cloning, some N+1 pools, some Erasure Coding pools, some physical hardware and some virtual machines, some Docker containerised services but not all, multiple frontends hooked together by password based SSH, and no management will to invest or pay for support/consultants, some parts running over IPv6 and some over IPv4, none with DNS names, some front-ends with redundant multiple back end links, others with only one. A well-designed, well-planned, management-supported cluster with skilled admins can likely run with finer tolerances.

Disagree. I run Ceph in Proxmox and have for years on a small cluster of 3 used R620 servers without any SSDs.

It’s just worked. I’ve lost two of the machines due to memory failures at two different points in time and the k8s clusters sitting on top didn’t fail, even the Postgres databases running with cnpg remained ready and available during both hardware failures.

Oh sure it works, not denying that. My point is that performance isn't great and if you only have a small cluster then it doesn't take much to make everything fall over because your failure domains are huge (in your case, you only have 3).

But then to offset the above, it also depends on how important your environment is; homelabs don't usually require five nines.

I am a big Proxmox fan but I dislike how easy it makes Ceph to run (or rather, how it appears to be easy). Ceph can fail in so many ways (I've seen a lot of them) and most people who set a Ceph cluster up through the UI are going to have a hard time recovering their data when things go south.