I am consulting with a startup right now that chose to go everything docker/k8s. The CTO is half-shocked/half-depressed by the complexity of our architecture meetings, although he used to be a banking software architect in his previous assignments. Every question I ask ends up in a long 15 minutes monologue by the guy who architected all of it, even the most simple questions. They are soon launching a mobile app (only a mobile app and its corresponding API, not even a website) and they already have more than 60 containers running and talking to each other across three k8s clusters and half of them interact directly with third-parties outside.
Even as I am being paid by the hour, I really feel sad for both the CTO and the developers attending the meeting.
k8s is definitely not for everyone. Google has thousands hardware systems running the same hypervisor, same OS, same container engine and highly specialized stacks of micro-services that need to run by the thousands. And even, I am not sure that k8s would satisfy Google's actual needs tbh.
Ironically, there are some companies that highly benefit from this and they are not necessarily "large" companies. In my case, k8s and devops in general made my life infinitely easier for on-site trainings: those who come with a poorly configured or decade-old laptop can actually enjoy the labs at the same pace than every other attendee.
If I count actually different containers (like, unique PodSpecs, or so), that count drops to ≈30. Even that is "high", and from an architectural standpoint, it isn't really a number I'd use. E.g., we have a simple daemon, but it also has a cronjob associated with it. So it has "2" PodSpecs by that count. But architecturally I'd call it a single thing. How it implements itself, that's up to it.
A lot of our "unique PodSpec" count, too, comes from utility type things that do one thing, and do it well. Logging (which comes from our vendor) is 3 PodSpecs. Metrics is another 3. We have a network latency measurement (literally ping shoved into a container…): PodSpec. A thing that checks certs to ensure they don't expire: PodSpec. HTTP proxy (for SSRF avoidance): PodSpec. A tool that rotates out nodes so that their OSes can be patched: PodSpec. Let's Encrypt automation (a third party too): 3 PodSpecs … but hey, it does its job, and it's a third party tool, so what do I care, so long as it works the API between me and it suffices (and honestly, its logs are pretty good. When it has had problems, I've usually been able to discern why from the logs). DB backup. But most of these don't really add much conceptual overhead; any one is maybe tied (conceptually) to our applications, but not really to all the other utilities. (E.g., there isn't really coupling between, say, the cert renewer and the logging tooling.) A confused/new dev might need to have it explained to them what any of those given tools do, ofc., but … many of them you can just Google.
… in previous jobs where we didn't use Kubernetes, we mostly just ignored a lot of the tasks that these tools handle. E.g., reboot a VM for patches? It was a custom procedure, depending on VM, and what is running on that VM. You needed to understand that, determine what the implications were … etc. And the end result was that reboots just didn't happen. K8s abstracts that (in the form of PDBs, and readiness checks) and can thus automate it. (And ensure that new loads don't need TLC that … an app dev realistically isn't going to be given the time to give.)
If we needed a common thing on every node? That would be rough. We did finally get to having a common base VM image, but even them, all of the per-app VM images would need to be rebased on the newer one, and then all rolled out, and who/how would one even track that? And … in practice, it didn't happen.
The problem I see with this approach is that it has become very difficult to evaluate system-wide topics such as accessibility (or security, or performance) as we constantly deal with a very diverse technological stack and increasingly complex attack surface.
In my opinion, this makes finding competent people who can actually evaluate or assess work almost impossible, unless you hire a Lemming who will run some third-party scanner he found on GitHub: if the scanner doesn't say something is awful or critical, then almost everyone at the table is instantly convinced the system is perfectly robust.
I try to warn my clients by asking them if they think that a judge will be satisfied if they answer "we ran the scanner the other day and the scanner said it was all good" after a customer sues them for failing to comply with a disabilities act.
The problem is that its _not_ a google scale solution. Its something that _looks_ like a google scale solution, but is like a movie set compared to the real thing.
for example: https://kubernetes.io/docs/setup/best-practices/cluster-larg...
no more than 5k nodes.
Its extra ordinarily chatty at that scale, which means it;ll cost on inter-vpc traffic. I also strongly suspect that the whole thing is fragile at that size.
Having run a 36k node cluster in 2014, I know that K8s is just not designed for high scale high turnover vaguely complicated job graphs.
I get the allure, but in practice K8s is designed for a specific usecase, and most people don't have that usecase.
for most people you will want either ECS(its good enough, so long as you work around its fucking stupid service scheme) or something similar.
K8s is a "middle 80%" scale solution. Its not made to run Google (though Google uses it a ton internally). Its also not made for your average four person startup (though, if you've got that experience internally, its not a bad choice; its not Heroku, but its better than a lot of deployment options out there).
All I'd say is: I've worked in a "scale-up" B2B org under $1M in ARR. We were pretty monolithic; just a backend NodeJS app, and a frontend SSR React app, basic. Five engineers by the time I left. We used K8s (EKS+Fargate). Maybe 50 pods total, across two environments. It was fantastic. We never had to say No to any weird customer, product, or engineering decision which would be difficult in either more managed, or more legacy, systems. Customer wants a custom domain and'll pay $50k for it? Like five lines of YAML and update Route 53, done. Datadog sidecar container so we can ingest some APM traces? Ten lines of YAML copy-pasted from their docs, done. Update the cluster? Click one button in the AWS UI. Every developer wants their own staging environment? Ok, bit more work, but: create some namespaces, retool the CI a bit, we can deploy separate databases in there as well its only staging, actually pretty straightforward.
Half the stuff we did with k8s would have taken three times as long with more native AWS solutions, and some of it probably would have been impossible on something like Heroku. K8s strikes a balance. Its not the simplest thing in the world. I wouldn't grab it on day 1 of a startup's engineering journey. But I wouldn't knock a startup who does.
I can leave the cluster alone for weeks, and it'll take care of itself, my CI will build new docker containers, tooling will start rolling them out across the cluster, if deployments fail they get rolled back, and I get an email, etc.
At some point I was hands-off with the cluster for 6 months and everything kept itself up to date and running just fine.
The fundamental design just isn't infinitely scalable, and at a certain point, you might rather have some bulkheads/autonomy or regional diversity.
off the shelf: https://hradec.com/ebooks/CGI/RMS_1.0/rfm/User_Interface/Alf...
although that was with something like 6-10k nodes because there was an upper limit to how many dispatches alfred could do because it was single threaded, from the early 90s and not really designed to scale that high
https://renderman.pixar.com/tractor is probably what they use now, or https://www.opencue.io/
but any grid engine style dispatcher/manager will do what you want. It'll give you the primitives to manage wildly larger scale than k8s.
These clusters were on real steel, as elastic clusters were horrendously expensive, and the storage was/is nowhere near fast enough.
Nowadays, I'd use AWS batch, or at a push airflow.
An argument could be made for something like CodeDeploy being better integrated on ECS, but that's more of a "k8s doesn't need CodeDeploy but ECS might" kind of thing. And even then, I wouldn't touch it.
An argument could also be made that upgrading ECS clusters is a bit easier, as the cluster itself, uh, doesn't have a "version". But on Fargate, its pretty painless on EKS, and Fargate ECS tasks do have a "platform version" that generally doesn't have to be worried about (version: LATEST), but is nonzero nonetheless.
Which is really to say that both ECS and EKS puke complexity, because its AWS, but the volume is pretty similar.
[1] https://docs.aws.amazon.com/eks/latest/userguide/aws-load-ba...
Anywhere I’ve worked business always prioritizes high availability and close to zero downtime. No one sees a random delivered feature. But if a node fails at night - everybody knows it. Clients first of all.
We’ve achieved it all almost out of the box with EKS. Setup with Fargate nodes was literally a one-liner of eksctl.
Multiple environments are separated with namespaces. Leader elections between replicas are also easy. Lens is a very simple to use k8s IDE.
If you know what you’re doing with Kubernetes (don’t use EC2 for nodes, they fail randomly), it’s a breeze.
There are entire SaaS industries that could fit into a single Google/Facebook/Amazon datacenter.
Holy shit, is this considered small to medium enterprise now?
The estimated maximum single cluster is 300,000 containers.
That's pretty medium, I've ran more than a million processes before, and nomad has 1million containers as its challenge https://www.hashicorp.com/c1m
borg can handle this easily.
But I’ve been in situations where it would have been worthwhile. I’ve been in situations with 30,000 machines that needed to be controlled. Splitting them out into very many clusters would be a lot of wasted overhead in configuration, administration and because you lose nodes to masters.
With some exceptions, I believe that's a few orders of magnitude above what a small business can run on. Nowadays people just start their day by drinking some K2l-aid and spinning up a "basic" 6-node cluster for a development prototype.
Maybe I'm wrong, of course.
Forget a whole datacenter, even just one rack is an unimaginable amount of computing power, these days!
For almost all purposes, we don't really need thousands of containers running on unimaginably fast computers, coordinated by AI-driven automation systems. What we need is software that is not morbidly obese.
Is my observation correct that unicorns start to see that scale?
Folks, listen, if StackOverflow can run on this: https://nickcraver.com/blog/2016/02/17/stack-overflow-the-ar...
So can your doctor's appointment website, your little ML app or Notion clone.
"But...". No. You ain't gonna need it. Do some load testing, prove it to yourself. Now, multiply the load by 100x, reserve AWS resources and you're good to go.
And with-k8s and without-k8s to me is pretty similar: we vendor or FOSS most of it. The major cloud vendors all have container registries (of … varying quality…); similarly, at a previous company we used S3+a small shim as a Python package store. (We later moved to a vendored solution.)
ELK for logs meant having a daemon set up per VM. Easier in k8s where I can push a DaemonSet to the entire cluster. With VMs … it's a per-app nightmare, really. Even then, that's really not perfect. In practice, in both situations, I feel like you end up having to integrate the apps with the metrics/logs providers. There's just not a common format. Sometimes, there are some libraries, e.g., there's some stuff for Prom's HTTP metrics APIs. Logs … eugh. Nothing amazing; getting structured logging requires per-app changes regardless of what you do. Sure, in either VM or k8s, you can just "suck up syslog/journald / docker logs", but what format are those in? They're not, is the answer, and I find most places do a "one text log per line" assumption (and then have stuff with multiline logs that just gets destroyed/corrupted/lost by the logging daemon) and it misses out on any sort of structured logs. jsonlines through those channels is a slight step up, but usually requires app changes.
It's also a Google engineer caliber solution. Lots of teams said “hey if Google engineers do it then it must be good!”…but forgot that they didn’t have the same in-house talent as Google.
The API is the main drawcard of k8s in the first place, if you are off in ECS land all you are doing is wasting a bunch of time on a dead-end.
I would instead focus on getting to understand the basics of the API by using a hosted k8s service like GKE or EKS. Stick with some basic manifests, i.e deployments, services, ingress.
Once you have some stuff running you can start learning how it really works and goes together, i.e what are pods, why are pods immutable, why is a replicaset, how does a deployment orchestrate multiple replicasets, what are endpoints, what is the difference between pod readiness/liveliness.
Don't cheat yourself this early in the game, just learn things the right way from the start and save yourself a bunch of work.
If you want to stay on the Kubernetes route check out k3s. Super easy to setup and usable for small production workloads
I always thought that one of the advantages of going serverless was that you didn't have to worry about keeping the underlying operating system up-to-date. No needing to do a monthly "sudo apt update && apt upgrade" or whatever. But containers throw that all away when container images enter the world.
Instead of updating your operating system, you're updating your images...and it's basically the same thing.
Most would say even a server-ful system (k8s, or whatever) should be considered 'cattle not pets' with immutable nodes replaced as needed anyway. No update, just replace. Just like building a new image and having new pods (or serverless whatevers) pull it.
I’ve never cordoned off an individual head of cattle and lobotomized it, which is kinda what we do when debugging issues. We take the pod out of rotation, flip a bunch of configs, then give it some traffic to see the new debugging statements.
If you're truly "serverless" by my interpretation of it, then you wouldn't care. Your cloud provider will have updated their infrastructure, and that's all that matters.
In fact almost nothing is serverless (well, the truth comes out! ;)) by that definition, since even Lambda has runtime versions to choose/upgrade, Managed-Acme has Acme versions, etc.
SES, SNS, SQS, etc. sure, but I suppose no compute, since you need libraries, and libraries have versions, and you can't have them (significantly/major versions) changing under your feet. (Or if they don't have versions they're of course destined to have known security holes.)
(Or it's not even about libraries if you want to say no you don't need libraries - it's just about having to interface with anything.)
Before I used Kubernetes for my side projects and only at work I always thought it's hard to operate and very tricky. If you start with an empty "default" cluster and then just add bits when you need them it's actually not that complicated and doesn't feel too heavyweight. I'd suggest to just play around with a simple example and then see how it goes.
There's things that are used in "production" clusters that you don't need at the beginning, like rbac rules, Prometheus annotations etc.
If you want to learn about containers, distributed workloads, etc, then Nomad is a great option that is easy to learn/adopt piecemeal.
There's now even onprem ECS variants which means not having to pay aws very much and still get the benefit of them running and maintaining the control plane
It's ridiculous overkill, but I'm looking at a NextCloud server on one of their PiBox hardware servers for the house. (You don't need a PiBox - their stuff will run fine on little instances from AWS/DigitalOcean/Hertzner, etc., or a spare PC you have lying around...)
Regarding PaaS stuff like Heroku, the only people I know that are still using that are solo hackers.
If you need less than 8 instances to do host your product, run far away anytime anyone mentions k8s