This kind of resilience is a form of art, and it's also kind of a full-time job.

I would not advise trying this for a "side project at work".

Generally we all agree that we move to the cloud and it's "fully managed". If it goes down - that's the price we pay.

If you have to ask the question "what if the RDS goes down", then you are really in a different universe.

That last guarantee of uptime requires a ton of work, testing, and money, because you are all the way up and to the right on the curve of diminishing returns.

My company also runs on a k8s cluster, but I agree with this fully and I keep my DB outside the cluster. For me, the order of preference is: RDS, Cloud Instance, Bare metal DB server, something else that is not kubernetes.

RDS will fail, but it likely will come back up without much action on my part. K8s will fail, you will spend untold human hours figuring out the k8s failure modes, before figuring out the database failure modes (which likely are quite straight forward). It's just a cost that is not worth it.

> If you have to ask the question "what if the RDS goes down", then you are really in a different universe.

It does go down though, don’t neglect the possibility because it likely will happen. With very average workloads, I’ve seen RDS databases restart unexpectedly, read replicas being completely out of service, and even databases being completely frozen (can’t even connect as root).

I’d still go with managed, but it certainly doesn’t give full reliability :) you still have to consider “what if it goes down” - it will!

Problem is that RDS comes at a price. It is purely about operation cost.

When you have 1500+ databases these cost add up. At that point, this kind of techniques are required to self host the databases. Price per DB with HPA na VPA is way lower than what you would pay for managed databases as well as you can hire a full time devops+dbadmin and still be cheaper.

This is a perfect example of a cost savings opportunity that offers no benefit and is really just a foot gun when it comes to operational complexity. The cloud cost savings almost never justify the operational costs here. There’s typically a very long list of other cost optimizations to be made before this would ever be on the table for me. If you’re talking about 1500+ databases and think a single full time devops+dbadmin in some kind of unicorn person is going to be a better option, I feel bad for both of you.

Except the typical cloud prices are dialed for USA, and it's not that unusual in my experience for the cloud savings to be high enough to add more than one highly skilled engineer who will give large positive impact all around.

I am not implying hiring single full time dbadmin, you can hire a team of them and still be cheaper that what AWS would cost at that scale.

> That last guarantee of uptime requires a ton of work, testing, and money, because you are all the way up and to the right on the curve of diminishing returns.

Not when it's a core part of your business...?

Not really. Those are two areas you need to really understand when things go south.

In Kubernetes you "solve" a lot of application complexity by abstracting it into things like Helm Charts or Operators. It's fun and easy to use operators to deploy databases, monitoring, mesh, minio... This does not make complexity disappear, it just abstracts it by creating another layer with an interface (k8s API) you may be more familiar with.

Whole Kubernetes in practice is adding more and more layers on top of it. All those layers/tools are usually amazing but soon you need some real k8s skills AND some app skills to handle even unavoidable scenarios like ... upgrades.

That's why it's so confusing sometimes and has such diverse opinions on. There aren't many environments that run vanilla cluster...

In the case of running Postgres on K8s, the problem arises immediately when you try to resize a data volume and you can't because the API doesn't support it. K8s is not really for stateful systems, yet, and systems like Postgres that prefer to manage their own resources, you don't want another layer which doesn't cooperate to get in your way.

"K8s is not really for stateful systems"

As a relative novice in the space, I'm grateful to hear someone say this out loud. K8s seems perfect to me for quickly scaling transient stuff like pipeline workers, web servers, but I've always been pretty leery of giving up the trivial snapshotting and rollbacks and other creature comforts of old-school virtualization when it comes to deploying long running applications, databases, and so on. And I've always felt kind of kind of guilty for not being on board to just mindlessly k8s-all-the-things.

Our company is finally looking at containerization and orchestration, and one product had the gall to say, Docker isn't for us. The non-technical people gasped because all the other products are moving towards orchestration!

Why? it's an ancient Windows Client/Server app. Each "node" manages its own state and communicates with each other in this proprietary, janky-ass way. It takes 10 minutes to start up a node.

K8/Swarm isn't going to do squat for this team except maybe launch dev/test environments a little easier.

>but I've always been pretty leery of giving up the trivial snapshotting and rollbacks and other creature comforts of old-school virtualization when it comes to deploying long running applications,

You can do that too with k8s with APIs which support more than just one backend.

> ...the problem arises immediately when you try to resize a data volume and you can't because the API doesn't support it

What API doesn't support it? k8s has support for resizing PVs and has for a while now. AFAIK all 3 cloud providers (and more) support increasing the PV using their storage class.

> K8s is not really for stateful systems, yet

Is this written somewhere or is it just your opinion?

Here is an announcement from last year stating that volume expansion is stable since 1.24: https://kubernetes.io/blog/2022/05/05/volume-expansion-ga/

The zalando postgres-operator also mentions as a feature:

> Live volume resize without pod restarts (AWS EBS, PVC)

What about changing iops or storage class?

I don’t agree here. There are operators like the one I’m a maintainer of (CloudnativePG) which works directly with Kubernetes, teaching it how to handle Postgres clusters as a coordinated set of instances. Enormous improvements have been done in the last couple of years, and we are particularly focused in working together with storage groups in Kubernetes to handle database workloads, such as for example declarative support for tablespaces and volume snapshots.

I suggest you to read CloudnativePG documentation as well as this article: https://www.cncf.io/blog/2023/09/29/recommended-architecture...

Also watch the video of my talk at last Kubecon in Chicago about handling very large databases.

I hope this helps.

Thanks for working on CNPG!

I have been a very happy user of CNPG even with occasional issues (database backup to GCS tripped me few times, but it works - mostly a bit of UX that I never was sure wasn't some fail of mine).

Now I only really need to add some automation for handling "recover the database and switch over clients to it" that is more automated (I understand why CNPG doesn't do recovery to existing database, but it is a bit annoying)

Volume resizing support has been added since ~1.11 which is like 5 years old at this point

I am the author of k8s resizing feature and its been GAed for awhile and feedback we have got so far has been good. If anything running inside k8s makes it relatively easy to support resizing. You just need to specify new size for PVC and it will both perform resizing on the cloudprovider and of the file system (if needed).

Modifying IOPS and other volume attributes is something less frequently needed but we just released alpha support for that too, if you must need it.

We have also added support for reporting volume usage in CSI specs, which I know some operators use to automatically resize volumes when certain threshold is reached (I however do not recommend using ephemeral metrics for automating something like this). But point is - you can actually define CRDs that persist volume usage and have it used by an higher level operator.

Another thing is - k8s makes it relatively easy to take snapshots which can be automated too and that should give someone additional peace of mind if something goes haywire.

Obviously I am biased and I know there are some lingering issues that require manual intervention when using stateful workloads (such as when a node crashes), but k8s should be just as good for running stateful workloads IMO.

Another thing is - k8s volumes are nothing but bind mounts from host namespace into container's namespace and hence there should be no performance penalty of using them.

Is the complexity better or worse than alternatives? What are the alternatives?

People use complexity as a boogieman to justify throwing together their own really wild chaotic & only so-so tested "simple" alternatives all the time.

To me, this feels like a modern wonder. We have layers of responsibility. Many people operate Kubernetes clusters already for all kinds of reasons. It provides a powerful broad base. Now on top of that, we can operate Postgres, with a very smart failover system that's super well tested & broadly used, that leverages this competent starting place.

What are the Fears Uncertainties and Doubts you have that make you scared about composition? What would help address specific concerns? To me, this division of responsibilities & use of consistent platform for a variety of needs feels like a huge win.

People love "simple" options but they're not. Run naked through the woods like savages option has appeal, but just getting started keeps adding up:

Sure, just add some bash scripts for some wal backups that go off-site. Easy! Install pgbouncer like one does, just a quick install, point it at the right systems. Setup some replication. Install and configure more kind of hairy software to make it HA. Configure some TLS cert yourself to not send naked traffic over wire. Add monitoring!

Then operationally, how quickly do you think you'll be able to fail over (with ogbouncer staying ok), failback, do an upgrade, add more replicas, lose a replica? Can you rotate cert reliably in a timely fashion? Can your replacement? How well did you document everything? Will you have all the monitoring you need when incidents start coming in, or did you just spitball a couple metrics into place?

The alternatives, in my view, are obviously bad. You can do them. Either cheaply with risk or industriously with effort & applied-talent. But having cohesive autonomic systems at our back that try to help, that can faultlessly do many common tasks with perfect accuracy (across unimaginable numbers of systems, with perfect consistency, in record time): that feels like a massively better place in the universe, one that I don't get why so many people kick scream & drag against. Rarely are their arguments well elaborated ("scary"), and their counter-suggestions feel like they massively underrated how multi-faceted & carefully connected production systems are, for good reason, and how hard it can be to remember to not forget to change X when you do Y.

> What are the alternatives?

Installing the Postgres on the computer, or on a VM.

Really, just the fact that some people keep asking this question is enough to question everything else they say. The alternatives are obvious.

Exactly. The way people successfully did HA and scalability on databases for more than 20 years before Kubernetes existed.

With a bunch of hand crafted, hairy code to link the necessary features like fail over, backups, PITR, clustering.

I know, I did all of that.

Which is why I punt that effort to a k8s operator these days, because at the end of the day it does everything I did manually plus makes it easier for me to spin a new database that has WAL shipping and backups to different location.

If you're punting to an operator why k8s? There are lots of managed database providers and if they're managing it, then k8s (or not) is an implementation detail for them to worry about.

Lots of various reasons. Performance, cost, compliance, and reliability are the ones I've personally seen.

These are each specific DB-as-a-services. Instead of sinking my time into learning something universal & competent & capable at many things (Kubernetes) now I'm investing in learning & using a niche service, that I won't have fine control over & whose limits might only be clear down the road. That's a risk.

Sure the dbaas might have superb autoscing and reliability. It may or may not have good OpenTofu/terraform providers available, or other custom niche deployment tools one would need to pick up & adapt.

Same questions as roughing it hacking together pg yourself, except now you have no power. Is observability going to be up to snuff? When performance problems come, how are you going to feel when you discover that analyst techniques a and b work fine but c and d can't be done because you are on managed service that doesn't actually give you full access to the db?

Costs can also add up, and are hard to predict. If you have a managed service, you'll be using a lot more network, which often has some cost (also some latency too). This isn't for everyone, but Kube makes it easy to start with anything (ex: RPi) and scale out. Works with hosted hardware with some local SSD, or I can get a 16-core 7950X and a bunch of gen 5 nvme drives and a fraction of a terabyte of ram for <$2000 and that'll scale me to godlike levels. Whereas I'll keep paying for something hosted.

I think it's so so ideal to have a reliably capable universal autonomic computing later underfoot. Most of the backend has had special specific answers to each question, each problem. We have been disjointed. Kubernetes provides a platform applicable to a vast range of workloads, where yes you need to tune and build different clusters for different workloads, but where the skills transfer much more readily. Everyone in Kubernetes knows how to ship resources definitions/manifests, and that base knowledge is all you need to learn to start consuming services - any services - on Kubernetes. Skill transfer is immediate. Having universal patterns (the API server), backed by autonomic operators making & keeping these resources real: it's so much better than the thousand different paths road of technology we've been on until now.

Because I'm not interested killing both performance (connecting to database over open internet), my opex (hello cloud bills!), and violating customer trust (one of the reasons they pay me is because they do not want to trust American cloud vendors), just so I can have slightly smaller complexity.

> Is there any way to overcome hidden complexity biting your hand other than studying k8s extensively?

No. Technology is hard. Operating services is complex, and gets more complex as the scale gets bigger.

Any attempt at making stuff simpler is usually either moving complexity elsewhere or making it more expensive (and in some cases, both).

I am happy using CockroachDB. The performance is not as good, since all your database writes require a 2 out of 3 quorum. But managing the database with the CockroachDB is pretty simple, since it can perform a rolling upgrade with no downtime.

Upgrades is handled with an operator, and happens by waiting all queries to finish, draining all connections, and restarting the pod with the newer version. The application can connect to any pod without any difference.

I perform upgrades twice a year, never really worried about it, and never had any availability problems with the database, even when GCP decides to restart the nodes to update the underlying k8s version.

I've never been a fan of deploying DBs, either traditional RDBMS or distributed, under containerization/orchestration other than quickly spinning up dev and test environments. Certainly not for production. Databases have been built for high availability for many decades. Sure, maybe RDBMS doesn't scale quite as easily as a node API application, but it's still not really rocket science and you usually have some leeway if you get a spike.

Adding Swarm or K8 just seems like redundant and unnecessary complexity.

Wouldn't most data driven, disk-IO intensive workloads be a bad fit for k8s? Especially when its not purely read-only..

That seems the crux of it.

This seems to be the common advise given, but I don't fully agree. There have been many times in my career where a DB was on a VM with the storage attached via a cloud providers block storage. When asked if we should move it to k8s, people are quick to mention k8s doesn't do well with persistent storage. However all of the big cloud providers offer the ability to easily create a persistent volume in k8s that then just creates a block device, attaches it to the k8s host and makes it available to the pod.

So in both situations you have the same IO limits of block storage. The question is does k8s persistent volume api add enough of an IO bottleneck to cause issues, IME that isn't the case.

Now if you want direct attached NVMe drives for higher IO than a network attached block storage will give, then it might be easier with a VM vs k8s, but I can't speak to that much.

Given k8s origin as bare-metal oriented system, attaching physical SAN volumes was there pretty early on, and now it's only more capable in that area (including passing devices for running exotic stuff)

Absolutely. I haven't gotten in the weeds of K8 volumes, but certainly have done a lot with Docker volumes including adding some PRs to fix some issues I've ran into. There is a lot under the hood with Docker volumes. With how DBs like to optimize their IO operations, I'm frankly a little amazed how it somehow mostly works between the DB -> orchestrator volumes -> file system.

It works totally fine on k8s with iscsi/fcoe/nvme-backed volumes provided remote targets themselves offer sufficient performance for your db workload

K8s mostly drops docker volumes code (and many other parts of docker, TBH). You can pass a random mount point or even a raw device and it will have only the overhead of kernel IO code for namespaces, overhead you hit anyway.

The only "docker graph" stuff is in OCI container data itself.

My advice is pay someone to run the cluster, and you can use all that time you saved on k8s complexity to operate the stuff inside the cluster and save some money.

Features like k8s Operators are essentially professionally-written code that stands in for a human agent that would take actions such as managing DB nodes, renewing certs, and performing backups. If you use mature operators, you can save a lot of money for a bit of effort.