Re:Ask HN: If Kubernetes is the solution, why are there so m...
by AaronM on Wednesday June 01, 2022 @02:25PM
(#31586000)
We don't have an issue with that last point, lots of EC2 EKS nodes and they don't fail randomly. Were you using resources and limits correctly? EKS nodes can fall over randomly if you don't reserve resources on the nodes for system processes, and your workloads eat up all the resources. That's probably not well documented either.
[ Parent
| on HN ]
Re:Ask HN: If Kubernetes is the solution, why are there so m...
by Spivak on Wednesday June 01, 2022 @04:21PM
(#31587518)
EC2 instances are inherently unreliable and that's not a knock on them, that's exactly the contract that you get using them and you're supposed to plan your architecture around the fact that at any moment an EC2 instance could die. We lose about 2-3 EC2 nodes per day (not like our app stops, like Amazon's own instance health goes red) and we couldn't care less.
Re:Ask HN: If Kubernetes is the solution, why are there so m...
by jamesfinlayson on Thursday June 02, 2022 @01:08AM
(#31591557)
What percentage of EC2 nodes is that?
Re:Ask HN: If Kubernetes is the solution, why are there so m...
by theK on Thursday June 02, 2022 @04:36PM
(#31600192)
Empirically around 0.1%
Re:Ask HN: If Kubernetes is the solution, why are there so m...
by raffraffraff on Thursday June 02, 2022 @05:26AM
(#31592977)
Setting limits is important, but it always has been. Kubernetes nodes typically don't have a swap so without setting container limits, some critical process can OOM. With swap enabled, memory grows > pathological swapping ensures => caches get dropped making disk performance suck, and all the while your system is shuffling pages between memory and disk. So of course load hits 50+ and the machine turns into a 'black hole'. I've even seen a single VM do that, and cause so much disk IO that it took out the whole hypervisor (which had a single RAID volume)