In many environments, you don't actually need distributed storage:

- Application servers are stateless.

- All state lives in stateful databases.

- Those database handle high availability at the application layer by replication data, so you can use local storage and deploy them using StatefulSets. Has the extra benefit of avoiding the network storage latency penalty.

But StatefulSets require storage...someplace.

We're dealing with that exact issue right now. Our K8s cluster is not playing nice with a NetApp NAS. We are exploring other options such as local iscsi storage volumes ($$$) or an external state store db (slow).

The point is that with native HA, it doesn't have to be distributed storage. You can use locally attached SSDs or NVMe drives, which are straight forward to manage and fast.

Importantly, they also have independent failure modes. I’ve never known a SAN to be within an order of magnitude of the advertised uptime — and some interesting failures have been caused by HA features. Local storage avoids whole classes of problem and has much greater aggregate performance if you can make it work for your app.

I highly recommend Rook [1], which is based on Ceph, provides PersistentVolumes to k8s workloads and is also running on k8s itself.

[1] - https://rook.io/

I worked on Colossus at Google and Ceph is the closest thing out there. Gregory Farnum gave a great talk about it at the Open Source Summit 2017.

I heard from someone at a large company, though, that it's not getting a lot of love from Red Hat nowadays, even if you're a large paying customer. Now I'm curious.

I'm working on Rook+Ceph at Red Hat. Rook 1.0 was just released last week which added support for the very latest Nautilus release of Ceph.

Great to hear. I couldn't get more details from them, but it was something about both rebalancing and data recovery. They picked SwiftStack, which is at least in part open source. Maybe by now you have figured who that company is.

My team is still very interested in Ceph and Rook, for the record.

I'm curious also. I don't have any special insight into RedHat, but rook went 1.0 just last week: https://blog.rook.io/rook-v1-0-a-major-milestone-689ca4c7550...

That announcement gave me the impression the project is making active progress.

Really curious about Rook. The stakes of running something like that on K8S successfully seem quite a bit higher than stateless apps, and commensurately the rewards as well.

Big fan of rook myself. We took several nodes loaded with 16 disk drives and used Rook to make a Ceph and Minio cluster out of it. Great storage solution.

Have you done any disaster recovery exercises? We've started to play with it, but haven't gone that far yet.

Me too.

These days I usually recommend that any data an app wants to persist be written to an object store or database.

And that any app that needs to write critical data to a disk should be provisioned with config management on bare metal (or pods statically assigned to machines with attached storage if K8s is a must).

This is mostly due to how tough the problem of storage is on K8s, and how much is at risk when it goes south.

I have been using openebs (https://www.openebs.io/) at my work and its been pretty stable thus far.