I don't see how k8s avoids the need to learn about cloud vendor specific tech. e.g searching "aws RDS k8s" gives me a beta github packages and a bunch of blog posts on how to configure it right. It doesn't sound like much less work than learning how to use RDS without k8s - read their docs, figure out the API.
Maybe i'm an "old man yelling at new tech" but meh i just see very little value in k8s because you inevitably need to understand the layer beneath - linux (k8s is far from a non-leaky abstraction imo), PLUS all the complexity of k8s itself. I do see the value when managing a big and complex infra with 100s of servers or something, but very few people have that problem.
How do you run an application on a cluster of plain old linux machines? How do you do load balancing? How do you scale up and down? How do you update your app without downtime? How do you roll back easily if something goes wrong? How do you ensure all your servers are running the same version of dependencies? How do you update those dependencies? How do you replicate your environment if you want to add a new server to your cluster? If your app has microservices how do services discover each other? How do you mount volumes from cloud storage? How do you update configuration? How do you automatically restart failed applications? How do you monitor if your applications are working? How do you make sure the right number of MongoDB replicas are running at all times? How do you view your log files remotely? How do you port-forward from localhost to your Linux server to test your app locally?
> How do you run an application on a cluster of plain old linux machines?
Build a package, install it in an image, run that image in an autoscaling group (or whatever equivalent your cloud of choice offers).
> How do you do load balancing?
An Elastic Load Balancer (v1 or v2), HAProxy, an F5 - this is deployment environment specific (just like in Kubernetes).
> How do you update your app without downtime?
Blue-green deployment, or phased rollout.
> How do you ensure all your servers are running the same version of dependencies?
Build them from a common image.
> How do you update those dependencies?
Update the Packer template that builds that image.
> How do you replicate your environment if you want to add a new server to your cluster?
Start the server from the same image.
> If your app has microservices how do services discover each other?
Consul, or DNS, depending on your appetite.
> How do you mount volumes from cloud storage?
It's a bit unclear exactly what you mean here, but I'll assume you mean either block devices (just attach them at machine boot, or on startup if they need a claim), or NFS.
> How do you update configuration?
Either update Consul and have it propagate configuration, or update a configuration package and push it out.
> How do you automatically restart failed applications?
Systemd restart policy.
> How do you monitor if your applications are working?
From outside - something like pingdom, and some kind of continuous testing. It's critical that this is measured from the perspective of a user.
> How do you make sure the right number of MongoDB replicas are running at all times?
Somewhat flippant answer here: the right number of MongoDB servers is zero. More generally, by limiting the size of an autoscaling group.
> How do you view your log files remotely?
Cloudwatch, Syslog, SSH (depending on requirements).
> How do you port-forward from localhost to your Linux server to test your app locally?
SSH.
Kubernetes attempts to do all of the above, which is why it's so massive, and I'd argue it's actually less complex than knowing all the tools above -- but it's an equal degree less universally applicable. In this way, it's perfect for the dev who never wants to "do ops", and less so for the dev that already knows ops (or any regular sysadmin/ops person), because they already know all these answers.
"Just use Packer, some AMIs, some ASGs, some CF templates, some ELBs, some EC2 instances!"
No thanks: I'll Terraform an EKS cluster in 30 lines of HCL and deploy my applications with a Dockerfile and a handful of YAML files.
Yeah, this is where it kicked in for me. Never mind the fact that all of that is AWS specific and absolutely doesn't help you if you ever move clouds. Great to know all the stuff below, but Kubernetes is a wonderful abstraction layer above that stuff, and it gets better every day.
CF could have become Kubernetes -- it was supposed to be, but it just never got the mixture right (and of course is AWS exclusive).
Yes. I would agree to your statement precisely as an answer to @jen20.
Some things such as getting stateful systems, HPAs and persistent storages were a little tricky initially but a breeze after.
But I do want to mention that you really really need a team to look after it. Without it, it'll bound to be another snowflake.
[edit]: <sigh/> i meant to say stateful when i wrote stateless.
Also, haproxy is one of the most reliable software I've ever used.
I wouldn't call it the best way; Rather a good way because Kubernetes does encapsulate the really good bits from scalability, development, security and reliability aspect. It's not a panacea but if you have team bandwidth to run k8s cluster, it's definitely worth a look.
<3 HAProxy. It's a solid piece of software tested to the teeth over the years and is great; Here's the thing: You can run it as your preferred ingress controller too :) https://www.haproxy.com/blog/dissecting-the-haproxy-kubernet...
If you were to deploy a solution like you described, you would get something more complex than simply running Kubernetes, except worse. I suspect that you believe your solution would be simpler only because you are more comfortable with those technologies than with k8s. The more I read criticism of k8s, the more I'm persuaded that what people calls "old boring technologies" truly is "technologies I'm comfortable with".
On top of that, you'd need to separately document everything you do on your infra. The advantage of Docker images over AMI is that you have a file that describes how the image was built. With an AMI, you would need to hope that the guy who created the AMI documented it somewhere (or hope that he has not quit). Same goes for k8s, where configurations are committed into your repository.
At the end of the day, k8s stays a tool that you should use only when needed (and also if you have the capabilities of using it), but I think you shouldn't discard it simply because you are capable of producing the same result by other means. You get a lot more by using k8s, in my opinion.
https://packer.io - how is this even a discussion?
> the more I'm persuaded that what people calls "old boring technologies" truly is "technologies I'm comfortable with".
My infrastructure runs in Nomad on bare metal. I am by no means opposed to “progress”, I just don’t think Kubernetes is the be-all-and-end-all of infrastructure and would like to have a less hysterical debate about it than the parent to my original post presented.
AMEN to this one.
Very nice answer. It's the way we run infra.
To be fair though, that's not "Plain old linux" like zaptheimpaler suggested was somehow possible. That's linux plus AWS managed services plus software from Hashicorp. Which is a great stack to be on, but has its own complexities and tradeoffs.
>> How do you run an application on a cluster of plain old linux machines?
> Build a package, install it in an image, run that image in an autoscaling group (or whatever equivalent your cloud of choice offers).
How come it is any different than running the Docker image on Kubernetes? The image build process is the same, delegating running the image to the platform is the same, the only difference at this point is the name of the platform you are running. Even if you were deploying ZIP archives to Elastic Beanstalk, if it doesn't work as expected, you'd have to debug it as an outsider, and you'd still have to know about the technology. I don't see how it is any different than Kubernetes.
>> How do you update your app without downtime?
> Blue-green deployment, or phased rollout.
How exactly? There are gazillion ways of doing them, they are rough concepts, what we need is a reliably working setup that requires as much effort from us as possible, and there are absolutely no standards on how to do them. Are you going to use Ansible? Maybe just SSH into the node and change the symlink? Maybe some other ways?
>> How do you replicate your environment if you want to add a new server to your cluster?
> Start the server from the same image.
How do you do that? You'd either do that manually on AWS console, or build some tooling to achieve that. If you were to do that via the autoscaling options the vendor is providing, then it is no different than Kubernetes: if that doesn't work then you'd have to debug regardless of the platform that is managing the autoscaling.
>> If your app has microservices how do services discover each other?
> Consul, or DNS, depending on your appetite
What is the difference between trying to learn how does Consul handle service discovery vs how does Kubernetes handle it?
>> How do you mount volumes from cloud storage?
> It's a bit unclear exactly what you mean here, but I'll assume you mean either block devices (just attach them at machine boot, or on startup if they need a claim), or NFS.
Would you mind sharing examples that are not vendor-specific and that'd be configurable on a per-service fashion easily, hopefully without writing any code?
>> How do you update configuration?
> Either update Consul and have it propagate configuration, or update a configuration package and push it out.
How is this any better than pushing your changes to Kubernetes? I personally don't know how does Consul work or how to update a configuration package and push it out to somewhere, I don't even know where to push them. In this context, learning them is also not any better than learning how to do them on Kubernetes.
>> How do you automatically restart failed applications?
> Systemd restart policy.
So, this means that you'd need to learn how to utilize Systemd properly in order to be able to start running your application and write the configuration for that somewhere, and also deal with propagating that configuration to all the machines.
>> How do you monitor if your applications are working?
> From outside - something like pingdom, and some kind of continuous testing. It's critical that this is measured from the perspective of a user.
The question was not really that. The tools like pingdom won't help you if an internal dependency of your application starts failing suddenly. You need a standardized solution for gathering various standard metrics from various services of yours, things like request rate, error rate, request durations, as well as defining custom metrics on the application level such as open database connections, latencies on dependencies, and so on. You will definitely need a proper metrics solution for running any serious workload, and in addition to that you'll also want to be able to alert on some of these metrics. There is no standardized solution for these problems, which means you'll need to roll your own.
>> How do you view your log files remotely?
> Cloudwatch, Syslog, SSH (depending on requirements).
The proper alternative to the Kubernetes' solution is Cloudwatch, and even then the simplicity of `kubectl logs <pod_name>` is still better than trying to understand how Cloudwatch works.
>> How do you port-forward from localhost to your Linux server to test your app locally?
> SSH.
This is not a trivial setup. Let's say you have a service A running remotely but it is not exposed, meaning that you cannot reach it from your local machine, and you'd like to be able to use that while developing your service B locally, how would you set this up in an easy way?
The points regarding the images are the same points as any Docker image, so it really boils down to the choice and one doesn't have an advantage over the other in this context.
What I am trying to say is: there are quite a lot of problems when running any kind of serious workload, and there are thousands of alternative combinations for solving them, and they were solved even before Kubernetes existed; however, there were no standardized way of doing things, and that's what Kubernetes is allowing people to do. There are definitely downsides of Kubernetes, but trying to point specific examples like these don't help as they are just names of individual software that also have learning curve and they all operate differently. I do wish there was a simpler solution, I wish Docker Swarm succeeded as a strong alternative for Kubernetes for simpler cases as it is brilliant working locally, and I wish we didn't have to deal with all these problems, but it is what it is.
As of today, I can write a Golang web application, prepare a 10-lines Dockerfile, write ~50 lines of YAML and I am good to go: I can deploy this application on any Kubernetes cluster on any cloud provider and have all these stuff defined above automatically. Do I need to add a Python application alongside: I just write another 20-lines Dockerfile for that application, again ~50 lines of YAML for Kubernetes deployment and bam, that's it. For both of these services I have automated recovery, load balancing, auto-scaling, rolling deployments, stateless deployments, aggregated logging, without writing any code for any tooling.
I agree with said parent. You can go a long way with ASGs, iptables and yum install - literally the entire first decade of your startup.
In larger companies, just the politics introduced with managing k8s by another dept. is mind-numbing.
And no, being able to run software is not equivalent to Kubernetes. It's not even in the same ballpark of the discussion. You are entirely free to go full Amish on your pet projects but let's not pretend that managing a pet server brings any operational advantage over, say, just being able to seamlessly deploy N versions of your app with out-of-the-box support for blue/green deployments and with the help of a fully versioned system that allows to undo and resume operating with a past configuration. You don't do that by flaunting iptables and yum, do you?
If anyone needs to operate a cluster then they need tooling to manage and operate a cluster. Your iptables script isn't it.
Read the actual post, instead of projecting your nonsense.
So yes, you'd need a handful of different systems, but "k8s" is hardly a single system once you have to actually manage it, and most of the parts are simpler.
Installing application and routing it with load balancing, TLS etc and support for draining is really simplified, something that used to require annoying amount of time (it got better as Lua scripting was extended in HAproxy, but by that time I also had k8s).
Resource allocation, including persistent storage (including NFS) is so much easier it's not funny, it makes my old years painful to think about. Syslog is not novel but getting all applications to log the same way was always a PITA, and at least with containers it's a bit easier (I still have to sometimes ship physical logs from containers...).
As for monitoring, that's one of the newer and more advanced topics, but it's possible to integrate more complex server monitoring with k8s - it already provides a lot of OS/application side monitoring that really makes it easier to setup observability - but now there's quite simple way to integrate custom node states into kubernetes allowing a quick glance way to check why a node is not accepting jobs, integrated with the system so you can actually trigger from a health-check that the node is in trouble and should not take jobs.
> How do you do load balancing?
With dedicated load balancers. They are even cheaper than servers because they are optimized for doing one thing.
> How do you scale up and down?
With pretty trivial automation than trigger deployments when load is too high.
> How do you update your app without downtime?
(app?!) A package manager pulls updated packages, deploys them and restarts the services. The LB does the flipping.
> How do you roll back easily if something goes wrong?
The package manager does that quite easily
> How do you ensure all your servers are running the same version of dependencies? > How do you update those dependencies?
Package managers support versioning on dependencies.
> How do you replicate your environment if you want to add a new server to your cluster?
You install MyStuff version 1.2.3 and everything goes in.
> If your app has microservices how do services discover each other?
Thankfully Amazon uses services that are not micro. DNS does the trick.
> How do you update configuration?
It's in the packages.
> How do you automatically restart failed applications?
OSes do that since decades.
> How do you monitor if your applications are working?
LBs check the endpoints, monitoring tools do the rest.
Essentially you can build all of this with the technology that existed 15 years ago. It works reliably and it takes much less learning.
Using plain old Linux and Erlang/Elixir as a backend you can get extremely far with scaling with very little overhead and issues.
It is unfortunate that so many ignore the benefits of using Erlang as a backend technology.
Thankfully, `jen20` was kind enough to take your questions in good faith and made a good demonstration of this confusion.
Also, see jen20's response.
If your only tool is a hammer (called k8s) then I'm sure everything really does look like a nail...
A handful of dedicated machines is enough. For example: stackoverflow.
None of these are difficult to answer. You can set up automated deployment/rollback however you want. You don’t care about the dependencies your code is compiled and linux has binary compat. You don’t need to split your app into 1000 unmanageable interdependent microservices. You have enough disk space on your dedicated machine or use NFS. You set up your systemd service file to auto restart your service. Etc etc.
When you have k8 as a hammer everything looks like a nail.
I guess if you pay for a hosted service a lot of the control plane is taken care of.
I’ve used k8s in orgs where it’s a great fit and really fills a need, but it is considerably more complex than running a web service balanced across a couple of machines, and it definitely requires a lot more upfront complexity (as in, you have to solve problems before you have they are actually problems).
Not sure what you mean here.
Kubernetes is designed to be cloud and vendor agnostic.
And RDS is a hosted database that you connect to from your application. Whether that applications runs standalone, in a container or VM is irrelevant.
But it’s not. Connecting to EKS is completely different from connecting to GCP. Setting up worker nodes is completely different too, oh and load balancers.
It’s only the last mile that’s similar.
Sure, it's far from drop & replace, but "completely different" is a huge misrepresentation in my experience. Running multi-cloud seemed quite doable.
To use k8s you need to know linux anyways, but to use linux you don’t need k8s knowledge. Most of things are as easy to setup with linux and the things where k8s really shines are not needed most often.
What I really wonder is where did you learn k8s, which parts did you learn the most? It seems huge and I would love to be in your position
For people who already know Linux, kubernetes comes naturally because it is pretty obvious.
But indeed, by experience, many companies can go to "unicorn scale" with two or three boxes.
Ansible can't give you anything like that. Even if you use Ansible to automate something like network setup, the commands and modules you use will be different between e.g. cloud providers. Kubernetes give you a consistent abstraction layer that's almost unchanged across providers, with the exception of annotations that are needed in some cases for integration with certain provider services.
For example, if you use just kubelet (the main daemon) without the api, drop some static yaml files (ala unit files) under /etc/kubernetes it will be something like systemd is. No big deal so far, but using the kubernetes api (just another daemon) it allows you to run the all the things anywhere.
I mean, you terminate one computer, k8s will figure out that the program should be running and it will be started in another computer. (Including moving detaching/attaching the block device from/into computers, this kind of things)
There is no much secret.
For me it really feels like an (well designed/integrated) API on top of standard Linux technology (ipvs, netfilter, mount, process mgmt, virtual ip, etc) via yaml running your stuff in 1..N computers but you manage it as one big computer :)
EDIT: added more context within "()"
To me as a developer heroku is the gold standard.
Docket-compose makes sense.
Kubernes and helm is a giant soul crushing wtf On getting anything done.
Now I’ll get it figure it out. Angular + exhaust+ typescript was a similar experience after I took a break from front end for a few years. A few months of pain, then it starts clicking.
but this just seems insane for a basic web app. So many different tools needed to get it going. Tutorials all have many steps that don’t apply, or use other odd pieces swapped in.
Oooold presentations (think 2016 and older) tended to talk more about basic building blocks and how they interacted, especially the design involving resources and their controllers working in a loop of "check requested state, check real state, do changes to implement requested state", and how those loops went up from single pods, through ReplicationControllers (now ReplicaSets), then Deployments, etc.
You had to become a master of figuring out the direction of where community was going and stay on bleeding edge.
Some people found this fun. I have many times enjoyed this. When I want to get a boring feature built and hand off to the jr develops so I can spend some time with my kids. It’s frustrating
The old presentations? Ones that remember K8s 1.0? There's one major change (move from Replication Controller to Deployment+ReplicaSet) that doesn't invalidate ~90% of the material, because the core stuff is about how controllers work!
Yet it seems more and more common to me that people don't learn the core mechanism of kubernetes unless per chance they got there writing CRDs :|
No. For starters, Ansible playbooks don't allow you to dynamically deploy and redeploy a set of containers in the node that at a given moment happens to have more available computational resources such as bandwidth, nor do teu respawn a container that just happened to crash.
Arguably Ansible only implementos a single feature of Kubernetes: deploy an application. Even so, while with Ansible you need to know and check all details about the underlying platform, with Kubernetes in essence you only need to say "please run x instantes of these containers".
I often get the feeling a lot of the negativity comes (rightfully so) from trying to replicate a current existing project into kubernetes. This is true of almost any paradigm - try replicating a Java EE monolith into Erlang and you are going to have a lot of problems. The big thing to note is that starting a project on Erlang very well might solve the problems that a Java EE project ran into, but that is because they were able to solve them at the ground floor, and just popping a Java EE project with all of it's architecture into an Erlang project will probably end up in a worse spot.
I think that this is what often happens with k8s as well - if you or your company have a currently working implementation that isn't on k8s, of course you won't be able to just easily plop it into a k8s cluster and everything be all well and good, but I think the problem is that people are equating that issue with k8s itself, which is a completely different paradigm.
And then tear your hair out when something doesn't work for some reason and root causing it requires learning a stupid number of layers. k8s is easy until it goes wrong.
If it doesn’t fail just right in kubernetes there might be no logs at all.
I’m thinking on particular about trying to mount filesystems into pods.
I migrated PCGamingWiki from running on some Hetzner boxes to DigitalOcean Kubernetes in a few days of work creating Dockerfiles and k8s manifests. I run a Kubernetes cluster at work fairly hands-free that hosts applications critical to our billing operations, and developers on the team deploy new applications with little or no support. Any of the issues I've hit are an artifact of migrating legacy applications not designed to run in more-or-less stateless environments, which is why the PCGW Community site still runs on its own server (Invision Community sucks).
I really don't see all the issues people have that aren't due to a mismatch of application design vs target environment (and no, it's not monolith vs microservice - monoliths run just fine on k8s; but you should be designing your application with a 12-factor environment in mind) or a misguided notion that you will be drowning in YAML hell (it's real, but you can manage it - and it's directly related to the complexity of the services you are deploying).
PCGW is a great example - installing a new Mediawiki extension, changing a config file, upgrading to a newer MW release is just updating a file or a git submodule and committing. I don’t get paid thousands a month to manage the site for the owner, so I make my time spent as efficient as can be done.
Hardly. If anything at all, it tells about the _team_ and/or the culture of the organisation. In any DevOps/SRE/Opsec culture worth the salt, an immediate blameless postmortem analysis would be performed to help with premortem analysis in future.
DevOps is not about exposing unsecured endpoints. You've got it all wrong son.
Fair enough. I am letting you know though the part where you got mixed up - that is not because _What DevOps has become in practice_. That is precisely because of failings and shortcomings in team culture and/or the organisation that practice DevOps.
But, if you're approaching it for the first time with no assistance, there are lots of things that can trip you up, and lots to learn. That's not a reflection on k8s, it's just the nature of the large set of problems it's solving.
K8s is succeeding because it's very well designed, has a large and diverse ecosystem, and solves set of important problems that very few other tools even try to tackle. Apache Mesos perhaps comes closest, but it's not quite as pragmatic, and its adoption level reflects that.
Also, because of k8s' scope, many people may not fully appreciate the range of problems it's solving, seeing it through the lens of their own background and focus.
This is true of software development in general, if not life itself!
But you're right that declarative systems amplify this issue.
I have done Softwaredevelopment for ~15 years, swiched now to infrastructure and build a gke cluster. It was awesome, very logical, nice and easy to use.
Now i read stories from coinbase and don't get it.
Paying for a managed provider. Heroku, Elastic Beanstalk, GAE, GCP, Fargate, etc. You push some buttons, they manage your cluster/services.
People still think they can get a free lunch by downloading some free software. If that were so, Windows and Mac would be dead and Linux would be the only desktop OS. But good news: I hear 2030 will be the year of the Linux Desktop!
Ahahahahaha, kids say the darndest things these days...
An inexpensive VPS runs compose well with more resources at a lower cost than the managed costs of k8s.
Kubernetes is designed for a cluster of nodes not a single one.
Of course a single VPS is lower cost and easier to use but what happens if you want a second or third. Or any form of redundancy.
Bonus points if you just mount the compose file on an NFS share.
I've wasted enough man months on kubernetes that unless you tell me to manage 1000 nodes this approach will never cost me more time than the time I spent using and learning kubernetes.
What is the alternative to k8s that does not need people to have any technical knowledge?
To me Kubernetes is extremely attractive because it helps me avoid learning cloud vendors' proprietary technologies. K8s is learn once, use everywhere, which is fantastic.
I am a 1-person venture doing everything from JavaScript/React to maintaining backend infra, and I couldn't have done it without k8s.