Manage Kubernetes Clusters on AWS Using Kops

Re:Manage Kubernetes Clusters on AWS Using Kops by yebyen on Friday July 07, 2017 @09:31AM (#14718159)

How are people doing autoscaling clusters on AWS?

I just want to create a simple, single-AZ cluster with one master, and one autoscaling group for workers. Ideally when there are no jobs to run, the ASG scales down to zero.

(I have a Jenkins service that can spawn pods, and it works great on the single-master-only cluster with no workers. There's even room for it to spawn one pod alongside it on the master node. I just removed the dedicated taint annotation. But as soon as I schedule one more pod, I wish I had an ASG for worker nodes because I'm requesting more CPU and RAM than is available, so stuck waiting for the first pod to complete and resources to free up.)

If I had more nodes (or the possibility of more nodes), I'd likely put the dedicated taint back and give my Jenkins master server pod a tolerates annotation so that it can land there when the resources are available, or use a label selector in the pod spec to ensure it winds up on the stable master, or both. I think I know enough to put this combination together, but I am sure I can't be the first, and I don't want to miss the "golden path" if there is one, this thread seemed like a good place to ask about it.

This seems like it would be a really common configuration, but I haven't been able to find a lot of documentation or even blogs about setting up ASGs this way.

I found kube-aws[1], which seems to be the only K8S orchestrator that actually mentions creating AutoScaling Groups for worker nodes in the setup docs (??). I found cluster-autoscaler[2] which appears to be the piece you need to get the cluster to scale up an ASG when unsatisfied K8S resource requests have left pending scheduled pods waiting, or scale down when nodes are idle for too long. I'm totally mystified why this does not appear to be a supported configuration of Tectonic or Mantl or Stackpoint, or... what's your favorite K8s orchestrator and how does it handle this?

[1]: https://github.com/kubernetes-incubator/kube-aws

[2]: https://github.com/kubernetes/autoscaler/tree/master/cluster...

[ Parent | on HN ]

Re:Manage Kubernetes Clusters on AWS Using Kops by philipcristiano on Friday July 07, 2017 @10:32AM (#14718598)

kops creates an autoscaling group for the worker nodes as part of the cluster creation process. Adding the cluster autoscaler is simple as deploying the autoscaler (as mentioned in your linked docs) pointing to the correct ASG. The IAM permissions for the autoscaler can be added with `kops edit cluster $CLUSTER`.

[ Parent | on HN ]

Re:Manage Kubernetes Clusters on AWS Using Kops by yebyen on Friday July 07, 2017 @10:49AM (#14718749)

Thanks! I have been using kubeadm for my "cluster" with some home-made ansible playbooks so I hadn't just gone ahead and tried it out yet. Good to know I'm headed down a path that others have already been down.

[ Parent | on HN ]

Re:Manage Kubernetes Clusters on AWS Using Kops by jdc0589 on Friday July 07, 2017 @11:12AM (#14718938)

I've spent a solid amount of time reading docs and looking in to this. It sounds like this might not be an issue for you, but the major gotcha is k8s does not support rescheduling pods when new nodes join the cluster yet to re-balance things (there was some stuff in proposal phase to address it though).

So, if your workload is made up of lots of short lived ephemeral stuff you are good to go, but otherwise you may have to manually step into rebalance stuff to new nodes.

The autoscaler addon might have addressed some of this, but I'm not seeing anything obvious after a cursory overview of the docs.

[ Parent | on HN ]

Re:Manage Kubernetes Clusters on AWS Using Kops by yebyen on Friday July 07, 2017 @12:38PM (#14719782)

I think I heard about that. Does sound like a pretty big gotcha, but I think you're right that it won't be a problem for my use case. Hopefully I can spend some time this weekend to try it out!

[ Parent | on HN ]

Re:Manage Kubernetes Clusters on AWS Using Kops by philipcristiano on Friday July 07, 2017 @12:51PM (#14719890)

The autoscaler will add minions when there is a pod that cannot be scheduled. It doesn't help in balancing existing things, but at least the new pod should be able to go there.

[ Parent | on HN ]

Re:Manage Kubernetes Clusters on AWS Using Kops by jdc0589 on Friday July 07, 2017 @12:56PM (#14719933)

I saw that and was confused, I thought that would work out of the box with k8s since (I think?) it continually tries to keep scheduling pods; I must be wrong though.

If balancing gets implemented, am I correct that it would probably happen in k8s core?

[ Parent | on HN ]

Re:Manage Kubernetes Clusters on AWS Using Kops by yebyen on Friday July 07, 2017 @10:32PM (#14723743)

I think I wouldn't assume that, myself; at least not after my stint in VietMWare (PTSD from a prior vSphere/vSAN experience)

We had VMTurbo/Turbonomic and VMWare's internal DRS that would sometimes compete against each other to decide where VMs should be scheduled.

You could handle balancing from inside of k8s core, or not. All you need is to evict the pod on the over-provisioned node, and to arrange for provisioning of the replacement node before the original pod is fully evicted. It should be the same for Kubernetes, if I had to guess.

[ Parent | on HN ]