Encrypting Postgres Data at Rest in Kubernetes

Re:Encrypting Postgres Data at Rest in Kubernetes by jandrewrogers on Sunday October 31, 2021 @03:42PM (#29059744)

Performance is a common reason. There are architectural incompatibilities between high-performance database engines and the way Kubernetes is designed to work. Kubernetes does not respect the traditional contract between the OS and the database engine, transparently interfering with that interface in adversarial ways that degrade important optimizations. Ironically, the performance loss can be substantially worse than running databases in a properly configured virtual machine -- virtual machines have some overhead but they otherwise don't interfere with the proper functioning of this software. Kubernetes wasn't designed to efficiently support the syscall and hardware utilization patterns of I/O intensive applications and this is evident throughout, even requiring non-standard hacks to set things up in Kubernetes that really shouldn't be necessary.

Deploying databases in Kubernetes is fine for many applications, I've done both. Not every application that uses a database is data intensive.

[ Parent | on HN ]

Re:Encrypting Postgres Data at Rest in Kubernetes by mindwok on Sunday October 31, 2021 @06:45PM (#29061207)

Do you have any specific examples of Kubernetes interfering with this contract? I've not heard of this kind of behaviour before.

[ Parent | on HN ]

Re:Encrypting Postgres Data at Rest in Kubernetes by jandrewrogers on Sunday October 31, 2021 @08:59PM (#29062036)

All high-performance database kernels do full kernel bypass e.g. they control their storage hardware, CPU affinity/context-switching, memory, network, etc explicitly. For all practical purposes Linux turns into little more than a device driver. This enables integer factor gains in performance via various optimizations. Ironically, it also makes the code simpler because behavior is explicit.

Linux is specifically designed to support this type of usage. The necessary syscalls were added decades ago, originally to support databases. Kubernetes intercepts these syscalls because they break its abstractions; while they appear to function like the underlying kernel syscall, the resultant behavior is not the same and generally unsuitable for these types of database architectures. The practical effect is degraded and unpredictable performance because it violates invariants that core optimizations rely on.

This has been kicked around by Kubernetes people for years, including within my own orgs because we use a lot of Kubernetes. No one has every been able to make this type of software achieve comparable performance, even when we've used a lot of hack-y workarounds. Kubernetes was not designed to allow software to interact with the Linux kernel in this way. Consequently, this type of software is deployed on VMs or bare metal in practice, even if everything else is on Kubernetes.

[ Parent | on HN ]

Re:Encrypting Postgres Data at Rest in Kubernetes by dilyevsky on Sunday October 31, 2021 @08:21PM (#29061815)

Sounds like parent misunderstood how PDs work in k8s or maybe is referring to THPs - still doesn’t make a ton of sense

[ Parent | on HN ]