News for hackers. Stuff that matters.
Re:TernFS – An exabyte scale, multi-region distributed files... by mgrandl on Thursday September 18, 2025 @11:47AM (#45291128)
There are definitely insanely large Ceph deployments. I have seen hundreds of PBs in production myself. Also your usecase sounds like something that should be quite manageable for Ceph to handle due to limited metadata activity, which tends to be the main painpoint with CephFS.
Re:TernFS – An exabyte scale, multi-region distributed files... by rostayob on Thursday September 18, 2025 @12:11PM (#45291405)
I'm not fully up to date since we looked into this a few years ago, at the time the CERN deployments of Ceph were cited as particularly large examples and they topped out at ~30PB.

Also note that when I say "single deployment" I mean that the full storage capacity is not subdivided in any way (i.e. there are no "zones" or "realms" or similar concepts). We wanted this to be the case after experiencing situations where we had significant overhead due to having to rebalance different storage buckets (albeit with a different piece of software, not Ceph).

If there are EB-scale Ceph deployments I'd love to hear more about them.

Re:TernFS – An exabyte scale, multi-region distributed files... by mgrandl on Thursday September 18, 2025 @12:41PM (#45291827)
There are much larger Ceph clusters, but they are enterprise owned and not really publicly talked about. Sadly I can’t share what I personally worked on.
Re:TernFS – An exabyte scale, multi-region distributed files... by rostayob on Thursday September 18, 2025 @01:05PM (#45292156)
The question is whether there are single Ceph deployments are that large. I believe Hetzner uses Ceph for its cloud offering, and that's probably very large, but I'd imagine that no single tenant is storing hundreds of PBs in it. So it's very easy to shard across many Ceph instances. In our use-case we have a single tenant which stores 100s of PBs (and soon EBs).
Re:TernFS – An exabyte scale, multi-region distributed files... by ttfvjktesd on Thursday September 18, 2025 @03:18PM (#45293809)
Digital Ocean is also using Ceph[1]. I think these cloud providers could easily have 100s of PBs Clusters at their size, but it's not public information.

Even smaller company's (< 500 employees) in today's big data collection age often have more than 1 PB of total data in their enterprise pool. Hosters like Digital Ocean hosts thousands of these companies.

I do think that Ceph will hit performance issues at that size and going into the EB range will likely require code changes.

My best guess would be that Hetzner, Digital Ocean and similar, maintain their own internal fork of Ceph and have customizations that tightly addresses their particular needs.

[1]: https://www.digitalocean.com/blog/why-we-chose-ceph-to-build...

Re:TernFS – An exabyte scale, multi-region distributed files... by mrngm on Friday September 19, 2025 @07:54AM (#45300571)
Ceph has opt-in telemetry since a couple of years. This dashboard[0] panel suggests there are about 4-5 clusters (that send telemetry) within the 32-64 PiB range.

It would be really interesting to see larger clusters join in on their telemetry as well.

[0] https://telemetry-public.ceph.com/d/ZFYuv1qWz/telemetry?orgI...

Re:TernFS – An exabyte scale, multi-region distributed files... by mdaniel on Friday September 19, 2025 @10:37AM (#45302155)
That was interesting, thank you. Other folks may also enjoy its sibling dashboard showing the used capacity https://telemetry-public.ceph.com/d/ZFYuv1qWz/telemetry?orgI...
Re:TernFS – An exabyte scale, multi-region distributed files... by kachapopopow on Thursday September 18, 2025 @12:00PM (#45291283)
Ceph is more of: here's a raw block of data, do whatever the hell you want with it, not really good for immutable data.
Re:TernFS – An exabyte scale, multi-region distributed files... by mgrandl on Thursday September 18, 2025 @12:40PM (#45291800)
Well sure you would have to enforce immutability at the client side.
Re:TernFS – An exabyte scale, multi-region distributed files... by kachapopopow on Thursday September 18, 2025 @02:57PM (#45293588)
It's more that it has all the systems to allow mutability which add a lot of overhead when used as an immutable system.