When you use Ceph in conjunction with Kubernetes, most people probably use rook. It's supposedly been production grade for a few years, but I keep running into random corner cases. In the beginning it would have problems when you ran it with XFS(the default at the time) causing the kernel to lock up.

A lot of the corner cases nowadays are related to not being able to scale the cluster, odd behavior with CRD's of users and other components that require you to run ceph admin commands directly through a proxy. The annoying thing about a lot of the "Cloud Native" projects is that you will spend a lot of time going through github issue lists.

There is a Taiwanese company that provides extremely cheap OEM ceph racks though. I've noticed that in terms of maintenance and performance something like that or a cheap TrueNAS almost certainly ends up cheaper than production workloads, unless you don't care about the data integrity of your ceph workloads.

I did have Problems with ceph many years ago but have many hyper-converged clusters that run ceph without any problems. In the last years I am running rook-managed ceph without any real problems.

When you say open source components, do you mean more commodity consumer-grade hardware or just the software components?