Your story with K8S sounds like a complete parallel of ours: we deploy a stack per customer and migrated from Chef. Would love to compare notes on how you're managing all the customer instances (we built in-house tooling layered on top of helm). Feel free to shoot me an email (in my profile).
Also built in-house tooling on top of helm.
How do you handle (if at all) network isolation between customers? I'm trying to find if there's a way to run one large cluster across multiple AWS VPCs.
Calico.
Also, my customers don't have strong requirements for network isolation. I'd be more concerned about compliance in areas like finance or healthcare.
Just seen your reply - thanks loads. Had not heard of Calico.
What's your team size now and when you first started using it?
Efficiency is increased a great deal. Previously, I'd have idle instances costing money when they were unused. Now my resource consumption is amortized across my entire infrastructure.
It also drives an architecture that lends itself to better availability. A pod in k8s may be rescheduled at any moment. If it's a stateless service, one may just increase the number of replicas to prevent a service interruption. If it's a stateful service, one is forced to think about how to persist data and gracefully resume operations.
It took a while for everyone on my team to get familiar with it, but once we started to really grok it, the sense of safety went way up. I feel pretty good about the fact that if someone accidentally blew away a cluster with thousands of pods in it, that I could easily replicate the entire thing and get it back up and running in tens of minutes without a lot of hassle.