When you install Kubernetes, you get a container runtime. That's a distribution I guess. Part of this seems like GNU/Linux.
The other stuff you're listing isn't solved by MRSK either...
Using local volumes is actually not an insane idea if your stateful service can handle data replication themselves: many modern databases can.
Obviously, you cannot move local volumes around.
And if you are setting up a database in Kubernetes... oh, you are in such a pit of troubles, that dealing with local volumes isn't really even worth mentioning. Surprisingly, your problems don't even start with storage, they start with memory. Databases really like memory, but use it very opportunistically, and scale well with load. So, when you configure your database, you tend to give them all the memory you have, but when they use it, it will really depend on the load and the kind of queries, how well they optimize it. Since Kubernetes scheduler doesn't really do well with reservations, you may run into situations where your database OOMs or just slows everything down, or doesn't perform well at all...
Next comes fsync. Unlike many unsophisticated applications, databases don't like losing data. That's why they want to use fsync in some capacity. But this creates problems sharing resources, again, well beyond anything Kubernetes can help with.
Next comes provisioning of high-quality storage for databases... and storage likes to come in the form of a device, not filesystem, but Kubernetes doesn't know how to deal with devices, so, it needs a help from CSIs of all sorts to do that, and depending on technology you choose, you'll have a very immersive journey into the world of hacks and muti-thousand page protocol descriptions telling you how to connect your storage and Kubernetes.
It might appear though, at the first glance, that things work well w/o much intervention, and there's a Helm chart for this or the other provider, and it's all at the tips of your fingers... but, as it often is in the world of storage, things get extremely complicated extremely quickly in case of errors. In such situations, Kubernetes will only obscure the problem. Oh, and errors in storage don't usually happen in the next hour or day or even year after you've set it up. It hits you few years later, once you've accumulated a ton of useful data and you've entirely forgotten how things have been set up, and folks in Kubernetes had moved on and broke stuff.
---
So, not only do you need small scale, you also need a very short temporal scale: don't expect your Kubernetes cluster to work well after about a year of being deployed. Probably not at all after five years.
But then... if it only works at small scale and for short time? -- is it really worth the trouble? I mean, Kubernetes isn't a small thing, it takes away a big constant share of your resources, which it promises to amortize with scale. You are essentially preaching the same idea as Electron-based desktop applications or Docker containers that create a lot of duplication of entire Linux user-space + a bunch of common libraries if you aren't extremely careful with that. Doesn't it become an argument for producing hot garbage as fast as possible so that someone else who can do a better job won't get a chance of selling their goods because they didn't have time to deliver?
>> Local volumes don't have a concept of quota. You cannot limit them to X bytes. So, if you give a single service a volume, it might just take the whole disk.
That's why we monitor our server disk for usage.
>> Obviously, you cannot move local volumes around.
Most of the time, this is not a requirements for database.
>> Since Kubernetes scheduler doesn't really do well with reservations, you may run into situations where your database OOMs or just slows everything down, or doesn't perform well at all
Unless it's test cluster with constraint resources, no other services will run on database nodes, through the use of taint and toleration. We can let the database use all the CPU and memory it wants
>> fsync Doesn't matter with local volume, since it's just a directory on the host system.
>> Next comes provisioning of high-quality storage for databases... and storage likes to come in the form of a device, not filesystem We didn't deploy our databases with raw block devices, even before K8S. Using regular filesystems make everything much simpler and we did not see any performance difference.
>> You are essentially preaching the same idea as Electron-based desktop applications or Docker containers that create a lot of duplication of entire Linux user-space + a bunch of common libraries
Yeah, no. If that's how you read it, be my guest, but don't put words into my mouth.
You are deeply misunderstanding Kubernetes if you think it's some sort of a turnkey solution that solves all your infrastructure problems. Virtually everything of value in Kubernetes isn't Kubernetes -- you have to add it on later, and manage it yourself. Container runtime? -- that's not Kubernetes. Database to store deployment info? -- that's not Kubernetes. Network setup and management? -- that's not Kubernetes. Storage setup and management? -- still not Kubernetes.
When you start using Kubernetes for real, you will end up replacing almost every component it brings by default with something else. CoreDNS? -- sucks for big systems. Volumes? You aren't going to be using volumes from local filesystem... that's insane! You'll probably set up Ceph or something like that and add some operators to help you use it. Permission management? -- Well, you are out of luck in a major way here... you have, basically, Kyverno, but it really, really sucks (and it's still not Kubernetes!).
Real-life Kubernetes deployments end up being very poorly stitched together piles of different components. So much so that you start wishing you'd never touched that thing because a huge fraction of the stuff you now need to manage is integration with Kubernetes on top of the functionality provided by these components.