I think this is a good answer. I'll add that as soon as you need to do something slightly more complex, without something like k8s you aren't going to be happy with your life. With k8s, it's almost a 1 liner. For example, adding a load balancer or a network volume or nginx or an SSL cert or auto scaling or...or...or...

Come on. What? Setting up a load balancer or nginx is considered complex now?

> What? Setting up a load balancer or nginx is considered complex now?

It's not complex to set up a load balancer in a given specific environment. But it's another kind of ask to say "set up a load balancer, but also make it so that the load balancer also exists in future dev environments that can be auto-set-up and auto-teared-down. And also make it so that load balancer will work on dev laptops, AWS, Azure, google, our private integration test cluster on site, and on our locally-hosted training environment, with the same configuration script." All of these things can be done in k8s, and basically are by default when you add your load balancer in k8s. They can be done other ways, too, or just ignored and not done, also. But k8s offers a standardized way to approach these kinds of things.

In my mind standardisation is how we really solve problems in much of software development. Everything before that is just practice and/or hacks.

> In my mind standardisation is how we really solve problems in much of software development.

I've been having this thought very often lately.

The only way for humans to do something faster is to use a machine. Any machine is built on some assumption that something is repeatedly true, that some things can be repeatedly interacted with in the same way.

Finding true invariants is very hard, but our world is increasingly malleable. Over time it is getting easier to invent new invariants and pad things out so that the invariant holds.

It's true not just for machines but engineering in general. Whether it's civil or mechanical or electronic or semiconductor engineering, their foundation is built on setting boundary conditions to make the natural world predictable so that it can be reliably manipulated. Things most often go wrong when those conditions are poorly understood, constrained, or modeled such as when using an unproven material, using imprecise parts, or ignoring thermal expansion when designing structural components.

Engineers have a plethora of quality control standards and centuries of built up knowledge to make this chaos manageable and the problems tractable.

I'm fine with standardization, as long as I set the standard. For standard 0, I propose that the only spelling for "standardization" will be with a 'z' (which is itself pronounced "zee").

American here too -- but don't think it's reasonable to demand American spelling from the rest of the Anglophone world.

I think the downvotes are missing the point. This is a key problem with standards: sometimes they standardize around something unreasonable. And tech is already riddled with all sorts of standards which are half-implemented in n different ways.

I think a better approach would be to have a specification for more robust negotiation protocols. When I see "standardisation," I already know that this means the same thing as "standardization" and furthermore that I should expect to see "colour"/"honour," organizations referred to in plural, "from today" rather than "beginning today" or "starting today," and even "jumpers" over "sweaters," "lorries" over "trucks," "biscuits" over "cookies," and more interrogative sentences in conversation. A British English speaker likely does the same process in reverse.

Perhaps I should clarify that "demanding American spelling is unreasonable" was specific to the context of HN (or other open discussion forums with international readership).

Within say, the volunteer-maintained documentation of MDN, the tradeoffs are quite different. There, ease of reading for reference by busy coders is much more valuable relative to ease of typing up a new contribution. Frequent switches between "color" and "colour" become a time-wasting distraction.

MDN should pick a standard and insist on it. And if Ubuntu chooses to require British spelling throughout, I'd say that's good.

Some of those requirements are far-fetched. Multiple cloud environments AND on-prem?

Ansible and Vagrant are not perfect, but I think they are far simpler than a single node k8s instance, and more representative of an actual production environment.

I’ve seen my company go multi cloud provider just to appeal to a single client. Now we’ll need multi cloud, multi continent setup to handle European clients. And I’m sure in another 2 years we’ll need our whole stack in China to support another clients requirements.

This is not my strength in any way, but hearing from those teams, Kubernetes will be a godsend

And one hour photo is now considered slow.

Uh, I have some news for you...

Setting up one is easy. Setting up one that gives multiple separate teams the ability to configure their services, and apply those changes to servers around the world and in different environments, repeatedly and safely, is harder.

We just spent months at my workplace working on a system to reliably define and configure a set of parallel silo'ed integrated datastores, services, and network stack within Kubernetes/ISTIO (and AWS), and to reliably upgrade new software revisions within those silos and to account for the changing "shape" of the configuration/content in these silos. It's repeatable and safe now, but it took a lot of effort.

Is any of your new stack open sourced?

Not yet ... perhaps some day.

There's huge difference between manually setting up load balancer - let's say HAproxy - and being able to just declare in an application that it needs "this and this configuration to route HTTP traffic to it."

The time I spent managing HAproxy for 5 services was bigger than the time I spent managing load-balancing and routing using k8s for >70 applications that together required >1000 load balanced entrypoints.

It's a lever for the sysadmin to spend less time on unnecessary work.

Depends who's doing the configuring. I don't know how to do it. But nor do I know how to use k8s for that matter.

Well, set me up one. :P

my pubkeys are below, send me an ip address and port to an ssh server

https://github.com/cameronnemo.keys

Anyone can ssh into a server and apt-get haproxy and tweak the configuration and get it "working" where the definition of working is accepting and routing traffic. But that's just a hobby setup. When people say setting up a load balancer is complex they are talking about professional setups, not a one off software install on a single server.

So give me a (possibly private) ASN, a couple bgp peers, and the frrouting suite. ACME is not difficult either.

But I want to be able to update my haproxy config with a git push, and roll it back with a single command, without sshing into anything, if something goes wrong. I want my everyday administration to be simple. Not the initial setup.

k8s did not invent gitops.

No but it sure as hell made it a lot easier and straightforward.

Now set it up in 30 data centers around the world, with the ability for dozens of different teams to add and change their applications, and across multiple staging and QA environments.

Why do I need 30?

1 rack in a datacenter is plenty for a database backed web app with a million users.

That is going to be killer latency for people in other parts of the world.

In my case, I work for a CDN, so we need to have data centers all around the world.

My what and what? :P I do get your point, it is "easy" for some definition of such, but to be fair, k8s would automatically put the ip and port in for my part of it all at least.

I never knew about this github feature. Nice!

You can use something like dokku for most of what you cited for much less overhead, if you are not planning to use the "highly available" feature for sure

Any competent engineer without experience is more likely to figure out how setup and/or troubleshoot nginx than kubernetes.

Just running docker-compose on load balanced machines is pretty close to having all k8s features (that would give you an endpoint, scaling, running pods[containers],heartbeats and nodes[vms]). If you run Kubernetes on GCP you will see it's just a wrapper of GCP vms, load balancers, instance groups and disks. EG: GCP k8's autoscaling for the nodes isn't any better than just simple GCP load balancers and instance groups (it literally is the same thing). k8's best feature (only?): specify yaml files to declare the setup. That is great! But, you make edits to this 4 times a year - that is a ton of complexity for those 4 git commits.

It’s pretty close to get it up and running. It’s not close to operating. You have to roll your own load balancer, health checks, method of updating and managing the underlying nodes, you don’t have operators or guarantees about the control plane, etc...

Spin up a K8S in any major cloud provider and you get all this with a consistent API, which is where the value lies.

You have to do all those things, minus the load balancer, with Kubernetes. It’s only if you don’t run on-premise Kubernetes that really start to se benefits.

Honestly, as an ops guy, I would prefer to get up at 3AM do deal with a failed VM or load balancer, compared to dealing with any kind of failure in a Kubernetes cluster at 10AM.

I can understand wanting to be able to deploy to Kubernetes, it’s extremely flexible and relatively easy. But managing and debugging Kubernetes is still a nightmare, even just monitoring it correctly isn’t exactly easy.

With Kubernetes, you can run multiple apps in the same cluster and let Kubernetes manage the capacity on GCP for you. Nice this you have shared headroom for all your apps versus an app per VM in your setup (I think?).

What is wrong with actually using Google’s hosted Kubernetes that would make you want to run compose yourself in their VMs and setup auto scaling, physical machine upgrades, etc.

You can replicate k8s featureset with plenty of other architectures.

But thats not the point.

k8s is the first infrastructure control plane in existence which is widley supported and standardized.

Can you get a managed docker swarm on aws, gcp, azure, do etc.? no.

How many kubernetes offerings do you know? I know at least 6.

Revolution doesn't need to be flashy and noisy.

Off the top of my head, deployments and sidecars are both missing, which are incredibly useful.

Change docker-compose to load balanced dokku nodes. Gives you the deployments and sidecars.

Dokku is magical. It blows my gob whenever I use it. It’s the best parts of Docker and Heroku together, and I can actually control everything that goes with my app.

I find that Dokku requires me to do too much on the machine itself, well as forcing me into a certain model.

I currently use exoframe with docker-compose files, and it's fantastic.

Haven't seen this one - it looks really good, thanks. I also like they integrated traefik in the solution.

Admiral gives you both with vanilla compose now.

Yeah, another tool, I know.

We edit yaml files everytime we update a service version, which happens multiple times per week? Is there a better way to deal with this?

Integrate it with your CI/CD pipeline. I have a Gitlab pipeline that runs kustomize and pipes the output to kubectl for every deploy to do things like setting secrets, image tags, creating ConfigMaps from config files in the repository, etc.

That's essentially what we already do. My original point was based on the fact that the config changes all the time (as opposed to 4 times a year).

What are you changing all the time in your config?

GKE uses other GCP products, but ”k8's autoscaling for the nodes isn't any better than just simple GCP load balancers and instance groups (it literally is the same thing)” isn't entirely accurate - GKE and K8S have logic that manages node pools that you won't be able to use with just instance groups.

OS for the cloud is exactly what it is. I see AWS, Azure and GCP as OEMs for cloud, just like Samsung, Oppo, Motorla, etc are OEMs for smartphones. Android was the open source abstraction across these devices. K8s is the open source abstraction across clouds.

The meaning of "app" on top of these two operating system abstractions is entirely different and the comparison probably doesn't extend beyond this. From a computing stack standpoint though, it makes sense.

AWS and other cloud provider provides far more reliable and simpler features/toolkings than k8s. For teams that's serious about building services on the cloud, having k8s to take over certain area of orchestration make sense, but the "operating system" K8s provides is just a tiny peace of the overall infrastructure.

Yes, people ought to do a side by side comparison of a new user learning to K8S v AWS v GCP before claiming Kubernetes adds more complexity than it returns in benefits.

Remember the first time you saw the AWS console? And the last time?

Hmm really? My experience is that, with k8s I end up learning all the complexities of k8s in addition to AWS.

Besides, personally I find AWS console much easier to understand. I don't get why people hate it.

> I don't get why people hate it.

Because it is hard to manage the configuration. It's why tools like terraform exist.

Anecdote. I worked for a small company that was later acquired. It turned out one of the long time employees had set up the company's AWS account using his own Amazon account. Bad on it's own. We built out the infra in AWS. A lot of it was "click-ops". There was no configuration management. Not even CloudFormation (which is not all that great in my opinion). Acquiring company realizes mistake after the fact. Asks employee to turn over account. Employee declines. Acquiring company bites the bullet and shells out a five figure sum to employee to "buy" his account. Could have been avoided with some form of config management.

> Acquiring company realizes mistake after the fact. Asks employee to turn over account. Employee declines. Acquiring company bites the bullet and shells out a five figure sum to employee to "buy" his account. Could have been avoided with some form of config management.

That is completely the wrong lesson from this anecdote.

1) The acquiring company didn't do proper due diligence. Sorry, this is diligence 101--where are the accounts and who has the keys?

2) Click-Ops is FINE. In a startup, you do what you need now and the future can go to hell because the company may be bankrupt tomorrow. You fix your infra when you need to in a startup.

3) Long-time employee seemed to have exactly the right amount of paranoia regarding his bosses. The fact that the buyout appears to have killed his job and paid so little that he was willing to torch his reputation and risk legal action for merely five figures says something.

Make sense. This is a classic example of bad operation management leads to unexpected outcome. Could have be solved much easier with proper configuration management.

Sounds like the exact nightmare a previous employer was living. AWS' (awful) web UI convinces the faint of heart to click through wizards for everything. If you're not using version control for _anything_ related to your infrastructure...you have my thoughts and prayers.

+1 for "click-ops", perfectly put.

> you have my thoughts and prayers.

Pretty much. The lesson learned for me was to always have version control for the complete stack including the infra for the stack. I like terraform for this. Terragrunt at least solves the issue of modules for terraform reducing the verbosity. Assume things could go wrong and you will need to redeploy EVERYTHING. I've been there.

Only 5 figures? That’s only what, a 100 hours of consulting?

Sounds like the tiniest acquisition mistake I’ve ever heard of.

Why not just migrate the resources?

Spin up new instances, load data from snapshots, get back to work.

Migration on resources taking live traffic is not an easy thing. Before migrating the traffic over to a different end-point, the time and the engineering work to make sure the new endpoints works is money and cost. Also there could be statefull data in the original account and to do a live migration of data with new data coming in at X tps is absolutely hard work.

Probably because paying 5 figures to the account owner was cheaper, faster and safer than recreating the production env from scratch

> Besides, personally I find AWS console much easier to understand. I don't get why people hate it.

The console is fine as a learning tool for deployment/management, and for occasional experimentation, monitoring, and troubleshooting, but any IaC tool is vastly more manageable for non-toy deployments where you need repeatability and consistency and/or the ability to manage more than a very small number of resources.

AWS console is very carefully designed to let you create traps that result in spending huge buckets of money if you're not paying attention.

EC2 instances are fairly predictable.

And when they fail (especially when the underlying hardware fails... Does AWS live migrate your workloads with zero downtime like Google?

Google can’t migrate if the underlying hardware fails quick enough.

I don’t think AWS has talked about live migrate, but given stability of their VMs and rareness of “we need to restart it notices”, it seems like they have something.

I've "experienced" a hard drive failure on both platforms.

On AWS, I was getting pagerduty'd because the solo gateway box was down, we couldn't ssh in so after an hour of no progress with dubugging or support, we just hit the reset button and hoped for the best. Fortunately this worked and later we where told there has a disk failure.

On GCP, I didn't even know and only discovered it in the logs when I was looking for other audit reasons. Turns out your long-running google VMs are being migrated all of the time and you had no idea. They actually have a policy / SLA around it, basically saying they refresh their entire fleet of servers every 6 weeks iirc. Honestly, if AWS is not doing something like this, I'd have increased concerns about leaky security neighbors (i.e. someone who has a VM running for multiple years without software updates. Hopefully you should be protected on shared servers, but it is software afterall)

Right, both vendors need to keep the underlying hypervisor and physical host up to date.

If you're only using EC2 you're overspending by a factor of 5-10. Compute is cheap.

Console is suppose to just be a web UI to quickly make a change or explore the feature. Any high quality engineering team should avoid make changes through console because it's not testable and repeatable. For code change, use codedeploy or container through ECS. For configuration and infrastructure changes, CloudFormation should be the right tool.

How do you view all the VMs in a project across the globe at the same time?

Do you need to manage keys when ssh'n into a VM?

Do you know what the purpose of all the products are? If you don't know one, are you able to at least have an idea what it's for without going to documentation?

The have also directly opposed many efforts for Kubernetes, even to their own customers, until they realized they couldn't win. Only then did they cave, and they are really doing the bare minimum. The most significant contribution to OSS they have made was a big middle finger to Elastic search...

Of course everyone's experience is different, but in my case...

> How do you view all the VMs in a project across the globe at the same time?

I'm not sure what it's got to do with k8s? I can't see jobs that belong to different k8s clusters at the same time, either.

> Do you need to manage keys when ssh'n into a VM?

Well, in k8s everybody who has access to the cluster can "ssh" into each pod as root and do whatever they want, or at least that's how I've seen it, but I'm not sure it's an improvement.

> Do you know what the purpose of all the products are? If you don't know one, are you able to at least have an idea what it's for without going to documentation?

Man, if I got a dime every time someone asked "Does anyone know who owns this kubernetes job?", I'll have... hmm maybe a dollar or two...

Of course k8s can be properly managed, but IMHO, whether it is properly managed is orthogonal to whether it's k8s or vanilla AWS.

If you're running pods as root, you're doing it wrong. That was a no-no with docker, and it's still a no-no for kubernetes. People still run non-containerized services as root too...

This is getting off-topic, but I didn't understand the rationale behind that. Processes running inside docker/k8s are already isolated, so unless you're running something potentially malicious, why would it matter if it's root or not?

(Of course, if you're running untrusted user code, then you'll need every protection you can muster, but I'm talking about running an internally developed application. If you can't trust that, you already have a bigger problem.)

If the container is running as root, and you escape the container, you are root on the host.

Containers share the kernel with the host, and are only as isolated as the uid the process in the container runs as and the privileges you grant that container.

He doesn't seem to understand the new landscape well enough to make his comparisons.

The point about AWS was not a Kubernetes comparison. It was a GCP one, because you asked what was wrong with the God aweful AWS

Docker won't run on my local machine as non root. And all the stuff I have seen so far has been run as root.

Beleive they were referring to AWS not k8

I've spent in total a tenth as much time learning k8s and related systems, as I have spent on AWS.

Most situations I have a direct comparison, k8s takes less ops. Often thanks to helm.

The AWS console is designed for lockin and I could use configuration management for AWS too but the time required to go through their way of doing x is just not worth it. Unless I want to become a AWS solutions architect consultant

Helm is becoming its own nightmare, fortunately work is already begun to replace it gracefully

Can you expand on this? We are just starting on Helm so curious to know your thoughts.

Sure, the most asinine thing (borrowing from Rob Pike) is to have a system where invisible characters define the scope and semantics of what you are writing. Now Helm takes this one step further (and I one beyond that before saying no more to myself and discovering https://cuelang.org) and starts using text interpolation with helpers for managing indentation in this invisibly scoped language. I hacked in imports, but was like, ok, making this worse.

So there's this problem and a number of experiments are going on. One camp has the idea of wrapping data / config in more code. These are your Pulumi and Darklang like systems. Then there is another camp that say you should wrap code in data and move away from programming, recursion, and Turing completeness. This seems like the right way to me for a lot of reasons both technical and haman centric.

I've pivoted my company (https://github.com/hofstadter-io/Hof) to be around and powered by Cue. Of the logical camp, it is by far going to be the best and comes from a very successful lineage. I'm blown away by it like when I found Go and k8s.

Recently migrated my company's k8s product to Cue and it's pure bliss.

Configuration should be data, not code. Cue has just the right amount of expressivity - anything more complex shouldn't be done at the configuration layer, but in the application or a separate operator.

What's the k8s product?

Are you on the newer syntax? #StyleDefs

It's not yet public - it's a new server OS for k8s. Think of CoreOS, but without a traditional userland and fully stateless.

And yes, we're on the newer syntax :-)

Just wanted to clarify that Dark isn't like Pulumi. The idea with Dark is that there is no configuration, just code.

There is still configuration, there has to be, you've just wrapped it so much it's not visible anymore (which is even worse than Pulumi, at least they are using an existing language). You still have to express (and write) the same information...

Darklang is solidly in the Pulumi camp, that's where outsiders put it. (I have seen the insides without beta / your demo, someone with a beta account showed me around a bit)

The real problem with Darklang is they have their own custom language and IDE. What exactly are you trying to solve?

> Remember the first time you saw the AWS console? And the last time?

There was a time in between for me - that was Rightscale.

For me, the real thing that k8s bring is not hardware-infra - but reliable ops automation.

Rightscale was the first place where I encountered scripted ops steps and my current view on k8s is that it is a massively superior operational automation framework.

The SRE teams which used Rightscale at my last job used to have "buttons to press for things", which roughly translated to "If the primary node fails, first promote the secondary, then get a new EC2 box, format it, install software, setup certificates, assign an elastic IP, configure it to be exactly like the previous secondary, then tie together replication and notify the consistent hashing."

The value was in the automation of the steps in about 4 domains - monitoring, node allocation, package installation and configuration realignment.

The Nagios, Puppet and Zookeeper combos for this was a complete pain & the complexity of k8s is that it is a "second system" from that problem space. The complexity was always there, but now the complexity is in the reactive ops code, which is the final resting place for it (unless you make your arch simpler).

> The SRE teams which used Rightscale at my last job used to have "buttons to press for things", which roughly translated to "If the primary node fails, first promote the secondary, then get a new EC2 box, format it, install software, setup certificates, assign an elastic IP, configure it to be exactly like the previous secondary, then tie together replication and notify the consistent hashing."

If I understand this correctly, all of the things could have been automated in AWS fairly easily .

"If the primary node fails" Health check from EC2 or ELB.

"get a new EC2 box" ASG will replace host if it fails health check.

"format it" The AMI should do it.

"install software, setup certificates" Userdata, or Cloud-init.

"assign an elastic IP, configure it to be exactly like the previous secondary, then tie together replication and notify the consistent hashing" This could be orchestrated by some kind of SWF workflow if it takes a long time or just some lambda function if it's within a few mins.

You could do that, but why? Just use k8s and move on.

Ansible. You can keep your YAML and deploy actual virtual servers on your cloud provider. Kubernetes is an introvert and this doesn't correspond to anything but it's padded cell walls. In Ansible it's an extrovert, an exoskeleton. Kubernetes insides out makes it right again. Make sense?

No, not following. I understand how master based provisioning management systems had their time and place, but we've largely moved beyond that to baking images, whether containers or VMs. Running a master based system comes with a whole house of other issues. Ansible is now relegated to a better system than based for installing packages and configuring the baked image. The time of booting a vanilla imstamce and then installing software when a scaling events happens is over.

By the way, what does ansible do to help with scaling applications?

Why do a comparison? K8S runs on AWS and GCP. They have managed services for setting up one. If you know K8S as a developer, then you simply consume the cloud K8S cluster.

I think the point is that there are people that claim that k8s adds a ton of complexity to your environment. But if you compare k8s alone with managing your infrastructure using (non-k8s) AWS or GCP primitives, you'll find that the complexity is similar.

The problem is that what the nodes in AWS are doing is fairly transparent. When my kubernetes pod does not come up, it’s always a hell of a pain figuring out why from just the events that kubernetes is giving me.

Sounds like you need better logmon or more practice. Debugging apps is so much easier on k8s.

While that's true on the managing instances side, you also need to actually deploy the infrastructure to manage them (If you're not using some PaaS offering). You don't need to do this for other IaaS.

Honestly the last time I looked at k8s was like 5 years ago, but back then it looked like a pretty big pita to admin.

The last 5 years have been transformative for both cloud native development and also open source software

It is a completely different world that stretches far beyond Kubernetes, though I attribute much of the change to what has happened from / around k8s -> cncf

It's so easy, I can launch production level clusters is 15 minutes with four keystrokes and make backups and restore to new ephemeral clusters with a few more simple commands

https://github.com/hofstadter-io/jumpfiles

(I'll be pushing these updates this weekend, haven't slept in 24 hours as reworked everything to be powered by https://cuelang.org )

> but back then it looked like a pretty big pita to admin

- well it's also a pita to update services without a downtime. - and it sucks to update operating systems without a downtime. - sometimes you reinvent the wheel, when you add another service or even a new website

however with k8s everything above is kinda the same, define a yaml file, apply it, it works.

and also k8s itself can be managed via ansible/k3s/kops/gke/kubeadmin/etc... it's way easier to create a cluster and manage it.

Exactly

That's exactly the point. You avoid lock-in to AWS or GCP by running on K8S instead. K8S becomes the "operating system": a standardized abstraction over different hardware.

Isn’t a “standardized abstraction over different hardware” just... an actual operating system? Isn’t “the operating system of the cloud” just... the actual operating system running on your cloud VMs? If you script a deployment environment atop vanilla Linux distros (e.g. ansible), you also avoid public cloud lock-in. (Side benefit: you also avoid container engine lock-in, and container layer complexity!)

Containers are a standard abstraction over the operating system, not over the hardware (or the VM, even). This has its use cases, but making it “the standard” for deployment of all apps and workloads is just bananas, in my view.

Kubernetes, when viewed as the OS for the data center, controls, manages, and allocates a pool of shared resources. When I install and run an application on my laptop, there are a ton of details I don't care about that just happen magically. Kubernetes maps this idea onto the resources and applications of the cloud.

Again, Kubernetes is far more than just deploying, running, mad scaling an application. It allows so many problems to be solved at the system level, outside of an application and developers awareness.

Take for example restricting base images at your organization. With Kubernetes, SecOps can install an application which scans all incoming jobs and either rejects them, or in more sophisticated setups, hot swaps the base image

I argue this is a good thing, like the Linux kernel and the Chromium kernel

What if instead they revamped their ancient microservices into modern monolithic applications?

I'm sorry, but I can't stand this kind of bullshit. You cannot possibly take two random things, put 'modern' in front of one word and 'ancient' in front of another, to justify changing things.

The problem of Kubernetes is probably that people started drinking the microservices koolaid and now need complex solution to deploy their software that became more complex when they adopted a microservices architecture.

IaaS is not "the cloud". it was in 2008 when all we had was EC2 and RDS.

Today Kubernetes is the antithesis of the cloud - Instead of consuming resources on demand you're launching VMs that need to run 24/7 and have specific roles and names like "master-1". Might as well rent bare-metal servers. It will cost you less.

You are completely wrong. It is trivial to have your k8s control plane and your workload cluster in something like an ASG, and basically know nothing about the nodes in either group. I'm not even using GKS or EKS, just ASGs on AWS. My servers and applications are cattle, not pets.

I knew literally nothing about k8s in September and now I have multiple clusters humming along, treating the worker cluster nodes as a generic pool of compute, autoscaling the cluster as well as the pods inside it. Upgrading is a breeze, I have great observability, I can deploy experiments and new applications with a single CI step or click, in fact I have nodes that are killed and get replaced for cost savings by SpotInst in the middle of the business day and I don't even need to know about it. My load balancers and even DNS are all provisioned for me and I can use the same Helm charts to create an identical staging and production environment.

Kubernetes IS the spirit of the Cloud and 12 factor apps. It's not that scary, and with tools like Rancher and k3s you can make it even simpler.

I don't have a horse in this nonsense but I find it hilarious that even users of Kubeanraenet have some trouble defining what it even is.

No trouble, it is a system that can do many things and has many optional pieces. It is fundamentally a cloud native orchestration system.

> Kubernetes IS the spirit of the Cloud and 12 factor apps.

I would argue that Kubernetes has _nothing whatsoever_ to do with the spirit of the Cloud, and that in fact "serverless" embodies the spirit of pay-as-you-use consumption models.

Doesn't kubernetes also have autoscaling capabilities?

https://kubernetes.io/blog/2016/07/autoscaling-in-kubernetes...

That is autoscaling within the cluster.... not the cluster itself.

Although, managed kubernetes clusters let you auto-scale the cluster itself, so i think the GP is wrong.

Check out Cluster AutoScaler: https://github.com/kubernetes/autoscaler/tree/master/cluster...

This tool allows you to autoscale the cluster itself with various cloud providers. There is a list of cloud providers it supports at the end of the readme.

I misread the link... it is genuine cluster scaling

> Might as well rent bare-metal servers. It will cost you less.

Long term, but up front costs are what make cloud services appealing.

FWIW, it's possible to minimize your idle VM costs to an extent. For example, you could use one or more autoscale groups for your cluster and keep them scaled to one vm each. Then use tools like cluster auto scaler to resize on demand as your workload grows. You are correct that idle vm costs can't be completely avoided. At least not as far as I am aware.

> > Might as well rent bare-metal servers. It will cost you less.

> Long term, but up front costs are what make cloud services appealing.

There are no up front costs. GP said rent dedicated, not buy your own metal. If there's anything in cloud it's the many pre-written services (queue, database etc) but GP is right: if you go k8s you aren't going to use many/at all so why not just go and rent cheap servers that get deployed in two minutes instead of renting expesive virtual servers which get deployed in a few seconds?

Does cloud really mean "resources on demand"?

Jonathan Schwartz, last CEO of Sun Microsystems, in March 2006:

"Frankly, it’s been tough to convince the largest enterprises that a public grid represents an attractive future. Just as I’m sure George Westinghouse was confounded by the Chief Electricity Officers of the time that resisted buying power from a grid, rather than building their own internal utilities."

https://jonathanischwartz.wordpress.com/2006/03/20/the-netwo...

If that is your definition of "cloud" then most stuff running on AWS and other "cloud" providers isn't. I agree that Kubernetes and even containers aren't the end all. I think they are a stepping stone to true on demand where you have the abstraction of just sandboxed processes compiled to WASM or something run wherever.

But as of where we are now, it is a good abstraction to get there. It provides a lot of stuff like service discovery, auto-scaling and redundancy. Yes you do need to have instances to run K8s, but that is as of date the only abstraction that we have on all cloud providers, local virtualization, and bare metal. So yes it isn't true on demand "cloud" but in order to work like that you need to fit into your service provider's framework and accept limitations on container size, runtime, deal with warm up times occasionally.

We had (have) discovery, auto-scaling and redundancy in PaaS. Most apps could run just fine in Cloud Foundry/App Engine/Beanstalk/Heroku. But the devs insist on MongoDB & Jenkins instead of using the cloud-provider solution and now you're back to defining VPCs, scaling policies, storage and whatnot.

Cloud foundry has been my best experience with enterprise cloud so far.

The issue was mainly with the admins not provisioning enough capacity, but for us devs it was fucking magical.

Knative does exactly that. You can use your buildpacks and everything in Tekton and configure your application via Knative Services. No need to bother about anything else.

> Might as well rent bare-metal servers

In light of this statement, what do you make of the fact that billions of dollars are spent on EC2? And of the people who spend that money?

Outdated. We were further ahead in the abstraction ladder with PaaS, and honestly most apps could run perfectly fine in Beanstalk/App Engine/Cloud Foundry/Heroku. But then the devs demand Jenkins, Artifactory and MongoDB, and instead of using cloud-provider alternatives you're back to defining VPCs and autoscaling groups.

Thats a whole lot of dangerous vendor lock in you've just listed there.

Exactly why we use ec2.

If AWS goes belly-up then we use terraform to make the equivalent on whatever new cloud provider pops up and keep going about our day.

No vendor lock-in

I read the entire article and couldn’t figure out what Kubernetes is. Your description helped a little.

Oh darn, I still don’t understand. Maybe I should learn what Docker is first?

"Approachable"

The buzzword mumbo-jumbo on the first paragraph alone (which isn't really even your fault or anything, just the bogus pomp inherent to k8s as a whole) is already a scarecrow to anyone that "wasn't born with the knowledge", really.

It is pretty hard to get used to it. Brushing it away won't make it approachable.