Whenever someone runs Kubernetes onprem I tell them to buy a TrueNAS or another cheap SAN. A cheap SAN costs as much as a DevOps expert setting up your Ceph infrastructure and a lot less when you actually run into issues with that software defined storage solution.

Once you do that Kubernetes is actually quite nice, because it gives you a base configuration of postgres that comes with automatic backup setup etc.

That advice is rarely taken though ...