Layer 1: bare metal servers, bare metal routers, hardware appliances, etc.
Layer 2: virtual servers, VLANs/security groups/etc, some appliances.
Layer 3: containers, container networks, storage resources, etc. Kubernetes, arguably Heroku.
Layer 4: functions as a service?
In the article the suggestion, or how I read it, is essentially "Kubernetes is nothing new, we've had [layer 2] for years". However I see Kubernetes as sitting squarely in "layer 3" here.
These layers are pretty subjective, all of the boundaries are blurred, but when I'm working these do require fairly different skills sets. I've never had the skillset to work at "1", I was fairly good at "2" for a while, but spent too much of my time working on that and too little shipping software, so my workplace moved to Kubernetes and it allowed me as an application engineer to do much less infra work, and for it to be at "3" when necessary.
To just write off everything in "3" as being "virtualisation" is ignoring the significant step in level of abstractions and the benefits brought by that.
Is your VM/node broken? It will heal itself because you throw it away and the new VM/node is fixed.
It enforces the abstraction of Service and VM. You will not install normal software on that VM just because you can. You don't need to give access to a VM to a developer who then needs root access and has dependencies and doesn't update the VM.
You no longer have dependencies to your VM because you can't have dependencies on your VM OS.
Abstracting it away from your VM also streamlines things like logfiles. You no longer need to collect all logfiles from VMs because you do it for your services and for your services, you only do it once (if even, log to stdout and be done with it)
Feel free to actually write more than 'None of this is true.' in a way that a discussion is actually possible.
Tx :)
> You no longer have dependencies to your VM because you can't have dependencies on your VM OS.
Your containers rely on the OS's kernel and whatever features it was compiled with.
> You will not install normal software on that VM just because you can
If you're paying through the nose for managed k8s, this is true. If not, you'll eventually need to login to a node and diagnose some issue, which means installing things on the node.
> You no longer need to collect all logfiles from VMs because you do it for your services and for your services
Whatever you installed to collect logfiles is getting them from the VM's disk (in /var/log/pods in k3s), unless your container is redirecting them somewhere that isn't stdout.
Managed Kubernetes on Amazon (EKS) is quite inexpensive: $0.10/hr * 24 hrs/day * 30 days/month = $72/month. Other costs are VMs, networking, and storage, which you would have allocated anyway. There are some downsides like forced upgrades, but cost is not of them for our use cases.
We incidentally don't ever login to Kubernetes nodes using tools like ssh. It's asking for security trouble to have those ports open.
The stuff that you still have to configure (e.g., firewalls, NFS) is all configured through a consistent, declarative interface (Kubernetes manifests) rather than a dozen bespoke, byzantine formats or imperative commands.
The declarative interface is going on the right direction (as far as yaml can be) but configuration management for it is still unsolved. Backups are also often forgotten; they're very easy with virtual machines.
> Out of the box, you get basically no isolation between anything
I'm pretty sure AWS Fargate and GCP's GVisor solve (or attempt to solve) isolation. Not sure about other cloud providers.
> you still have to deal with security contexts and have processes in place for keeping your container images secure
How do VMs help secure software artifacts beyond the security practices in the container ecosystem? And I would argue that "dealing with security contexts" is strictly better in Kubernetes than the equivalent in VMs if only because of the unified interface (Kubernetes manifests).
> If you use community Helm charts your services may end up running with essentially random privileges that may easily conflict.
You can run into the same issue with Ansible scripts on VMs. This isn't a Kubernetes specific issue--ultimately, all system administrators need to take care to run secure software on their systems. Neither Kubernetes nor VMs offer a silver bullet here.
> configuration management for it is still unsolved
If "configuration management" refers to configuration of the hosts, then yes, public cloud provider Kubernetes offerings solve for this--you don't have to manage the host configuration at all (unless you want to opt into it).
> Backups are also often forgotten; they're very easy with virtual machines.
The etcd backups are managed by the cloud providers, as are backups for mounted volumes. Not sure what backups you're thinking about.
Or 'can':
If you use a VM on AWS, you also need to know that you need to configure a vm snapshot (very easy, totally agreeing here with you).
But you can also use a managed k8s from AWS which you can also backup as they are all on PV and they have snapshotfeatures.
I don't want to compare a VM + Snapshotting 1:1 with kubernetes though. It wouldn't be fair to k8s and it wouldn't be fair for all usecases which work very very well on one VM.
And do you need to run this on Amazon, Gcloud, Azure,one of the smaller cloud providers like DigitalOcean or locally on Kind/k3s? It'll require very little work to get them working on any of these - if any. Cloud specific services and persistent storage will be the main issues, but that's something you can't really get around.
Now is it perfect? Absolutely not, as with any tech, there will always be problems and bottlenecks. But it allows development to scale, not just the workloads, and the skills required are transferable, which makes it a much easier sell.
> Cloud specific services and persistent storage will be the main issues, but that's something you can't really get around.
That isn't wrong necessarily, but products like OpenShift Container Storage (OpenShift Data Foundations now actually) can provide a common API to erase that problem. ODF uses Ceph under the hood so you can get block, file, and (s3 compatible) object storage no matter where you are.
Cloud specific services are indeed a problem, but many of them have open source/portable solutions that you an choose that can run everywhere. Such as Fission, RabbitMQ, Kafka (not my favorite), Argo CD, etc. Really the things I run into most now are things like AWS machine learning services.
It’s at the point of needing to scale horizontally where I begin to disagree with your premise. This is where you’ll typically get into proprietary and/or ugly offerings.
You still do with kubernetes
> network transfer speed, and many other things.
No, you don't. There's no slider for "network performance" on GCP, Azure or AWS.
Who said there was a slider? Network performance is one of the key filters on every major cloud provider. Not every instance type has the same network performance.
>You still do with kubernetes
Sort of. It can be way more abstracted away. What's the minimum I need to run a node? Okay great, now run as many nodes as needed, when needed. To set that kind of thing up with bare metal on AWS, for example, would require getting into some proprietary offerings and/or absurd complexity.
I had to actually go and look up with our default instance type was for our cluster. It's a rather useless fact since it doesn't much matter compared to the number of active nodes. That's not true at all if you're directly managing VMs.
I'd absolutely never trade the complexity of kubernetes for the complexity of self-managing a horizontally scalable bare metal VM implementation. However, some people (obviously) disagree with that. To each their own.
Here’s the thing: either you care about it (and you can game the sizing of the instance) or you dont and you run kubernetes.
But you if you don’t care then it doesn’t matter. There’s no slider for you to care about. It is not extra overhead.
But you configure the exact same things as with VMs and kubernetes.
Network performance (as per OP) is not configurable on either.
You just accept whatever accidental default you happen to have, it's not a conscious decision people are making, and it's an awkward assumption to say that you have to think about it.
Because if you have to think about it: that doesn't go away with kubernetes anyway: if anything it probably gets worse.
Even better: you define a “bandwidth” resource and each service requests a slice of that. Kubernetes takes care of the box packing. If you care so much about it you can then enforce that in a number of different, flexible ways depending on your infrastructure or requirements. At the end of the day it’s no different to CPU, memory or GPU requests.
You just maybe proved the point that I originally asserted: the things you configure on kubernetes are the same things you configure on cloud VMs.
Everyone who uses GPUs with Kubernetes does exactly this. GPUs are not a native thing to Kubernetes.
> You just maybe proved the point that I originally asserted: the things you configure on kubernetes are the same things you configure on cloud VMs
You are of course entirely missing the point, and I’m not sure if you’re doing it on purpose or not.
You have 100 units of work that you need to run. A unit of work is some “thing” that needs a certain number of CPU cores, memory, GPUs and other user-defined resources. Each unit of work also needs an individual identity, distinct from other units of work.
Go and code something to run that workload on the minimum number of cloud VMs, taking into account cost and your own user-defined scaling policies, minimizing the amount of unused resources. Now make it handle adapting to changes in the quantity and definitions of those units of work. Now make it handle over-committing, allowing units of work to have hard and soft limits that depend on the utilization of the underlying hardware. Now make it provision some form of secure identity per unit of work.
After you’ve spent time coding that, you’ll realize that:
1. It’s hard
2. You’ve re-invented part of Kubernetes
3. Your implementation is shit
4. It’s very much not “the same things you can configure on cloud VMs”
1. Kubernetes manifests require "requests" to be specified. (mem/CPU allocation)
2. Getting 100 VMs identical is not difficult on the cloud.
The point I'm making is that you've already abstracted a lot of the things away with Cloud, and we abstract the same exact things even more on top of kubernetes.
If K8S was running on bare metal I'd agree with you though.
This whole thread is discussing the "mental overhead" of managing VMs vs Kubernetes.
if you have to define the "size" of your workload, it hardly matters if it's a VM or k8s. You need to define the size.
Kubernetes can be more fine-grained (I want 1/4th of a CPU!) but you still define it.
I'm not talking about cost, or really anything, only that the original claim I originally responded to: "hurr durr but with VMs I have to configure everything!" but that is the same on kubernetes.
VMs are already a pretty good abstraction if you're looking at carving up compute resources. My "frustration" if I even have one is that we are doing both, one on top of the other. Which feels extremely wasteful.
But like everything it depends on your workloads, and I'm used to having things that consume entire CPU cores, not 1/4th of one. (I'm also not used to making web services these days, and kubernetes is optimised primarily for that kind of stateless workload)
Mixed workloads on smaller, larger machines are great for this.
With VMs you do need to configure everything, compared to a baseline stripped down AMI/image that runs nothing but docker and a Kubernetes daemon.
Yes, you can enumate Kubernetes with a bunch of custom tooling. No, it's not better. Yes, it is harder.
For example, if you use GKE (Google Cloud's K8s offering), you attach your K8s cluster to an auto-scaling node pool and it handles (de)provisioning of your VMs for you. You essentially don't care about the VMs, there's essentially no overhead.
If you are in a private cloud, this also creates a good "API boundary" between the team responsible for running hardware, and the team responsible for shipping software to run on that hardware. On the former side you can essentially just adopt a machine into the cluster and leave it, and on the latter side K8s lets you programmatically reference resources, but you don't need to know how/where they came from.
Yes, you don't need Kubernetes to come up with your own implicit or explicit API boundaries, and these might not be needed for smaller projects. I agree that Kubernetes is often used where it's not strictly needed.
There are things which strict abstraction, and with it, separation of concerns provide. The crucial point is that certain things are enforced.
You're right though, the abstraction layer is very interesting for _enabling_ public clouds, and for private cloud team ownership.
You also know that you can throw away VMs because they don't contain any state. You are not losing data just because you kill a VM or a VM breaks.
It is way easier to just spin up n nodes and provision them all equally than whatever you did before.
In my team, we can manage way way way more nodes than we ever could. We spin up 100 VMs and destroy them on a regular basis automatically.
Gardener for example supports autoscaling on bare metal. The whole ecosystem is providing tons of great options.
Lauding Kubernetes because ops work took too much of your time is just shifting your burden elsewhere, even if that means paying a bit more for an offering like ECS fargate.
Any environment with configuration management can treat instances as ephemeral. It’s a best practice.
I view docker more as a package manager, no more dependency hell.
In any event K8s is sprawling, it will soon be too complex for its own good. Assuming it’s not already.
I dont have anything against VMs. Feel free to click yourself a VM on any cloud provider, use it however you like.
K8s abstracts VMs away and i have and had real issues with maintaining VMs. Docker filling up the node with logs. Unable to upgrade the BaseOS due to python dependencies. Managing the same VM stack through ansible and everything ansible or chef brings to the table.
There have been plenty of self healing mechanism in place which do solve unfortunate issues. Memory? The service restarted, was offline for 3 minutes and is now working again. Node disk full? Pods get scheduled away, new node comes up, done. Update/upgrade of nodes? Nodepool does it for me.
For me, k8s has 2 real issues like memory (swap support is wip finally!) and stateful workloads like a database. But the concept of an operator shows a bright future.
k8s also does one thing very nice: It enforces certain aspects which are a pain in the ass later. That VM which wasn't updated for years and run just fine? Now there is an issue and it needs to be fixed asap. But now the debian repositories are no longer available. I have to fix apt srce list first, then i need to fix dependencies and then i need to restart it.