Kubernetes is incredibly daunting for someone who only used VMs, but after spending some time with it, it's not really that hard. I also unironically run my personal stuff on K8s just because it's familiar to me, and it is also a good exercise for a very useful skill. People complain about the price, but you can easily get a 3-node cluster on GCP for less than $15/month.

Yeah, I've been learning Kubernetes recently and a simple setup isn't too difficult. However, there's a lot of concepts involved with Kubernetes to grasp and I've definitely not learned everything there is. For example, Jobs still baffle me in regards to what is the purpose outside of CronJobs.

I use Jobs for helm hooks like integration testing and DB migrations associated with deployments. They work great for one off workloads like that.

You just beat me to it!

The term "batch job" (as opposed to "service") appears frequently in the Google K8s-like scheduler papers:

https://research.google/pubs/pub41684/ https://research.google/pubs/pub43438/

In other words, jobs seem like the go-to way to take advantage of unused CPU/memory.

I developed a simple wrapper around vegeta, called vajra, that can be given the URL to a file containing test URLs (in vegeta format), a target QPS/duration, and then easily run a load test.

Gets deployed as a k8s `Job` via GitLab (can be scheduled/on-demand), with a simple script echoing back the vegeta status every 1 minute. Jobs don't yet support automatic cleanup, so another simple GitLab job deletes the job from k8s upon completion. The execution log is anyway available in GitLab/Kibana.

All in all, an extremely simple way to run load tests against a service deployed in k8s.

What grew as a personal utility is now being used by many teams for quick load tests.

Jobs can get useful in more complex workflows. The most common use I’ve had for them us running a smoke test suite or a database migration as part of the larger workflow for a big Rails app.

But why wouldn't I just run a database migration as part of my CI flow? I don't understand why I would add a persistent element to my Kubernetes cluster for just a one-off task, since Jobs do not actually automatically remove themselves from the cluster. And as far as I know, you can't even reuse the already created Job, so if I want to migrate the database again, I need to make it from scratch.

Well, CI doesn't necessarily mean CD. In more than a few cases, I've found it advantageous to use CI to do automatic container builds, but manually attend to the deployment upgrades, particularly when things like db migrations are happening.

> Jobs do not actually automatically remove themselves from the cluster.

set `ttlSecondsAfterFinished` and they will

> And as far as I know, you can't even reuse the already created Job, so if I want to migrate the database again, I need to make it from scratch.

If you need to run a job on demand, you can always 'kubectl apply -f' the metadata, or you can create a cronjob and `kubectl create job --from` your cronjob. Or, if it needs to run at a particular part of your installation/upgrade workflow, use helm hooks. Jobs are intended to be a one-time thing, so the lack of functionality to repeatedly run them is intentional.

There's a TTLAfterFinished feature you can enable since 1.12. It's been in alpha forever. :/ Lets you add a ttl to your job spec to automatically clean up completed jobs.

One more unironic personal user here - I use k3s for cheap-as-possible.

I actually compared K3s with Docker Swarm in my master's thesis where i also wrote a simply Python tool to automate setting up servers for deploying containerized workloads on then.

While i can't really share the text (because it's all in Latvian), i did find that K3s is a pretty good option for when you want to use Kubernetes, but have pretty limited hardware resources. So much so, actually, that it used only slightly more resources than Docker Swarm (and, as a consequence, the deployments that were running on the nodes could serve almost equal amounts of requests under load).

Personally, i do believe that Kubernetes is sometimes overrated for simpler deployments, but if people don't mind the manifest syntax and are comfortable with the tool and its ecosystem, i think that K3s is probably the way to go in those situations (maybe with something like Portainer/dashboard running for graphical administration, if necessary). Such a good distribution, could probably even run it in prod with a HA setup.

> but you can easily get a 3-node cluster on GCP for less than $15/month.

Just wondering, how are you exposing your services from the cluster? (I assume you aren't using a load-balancer ingress)

Did you use Docker Swarm in the previous revision?

I've been quite happy with it for a few years now on a single-node cluster and a handful of services. I could easily add another node, but haven't had the need to. The setup was quite simple and it requires practically no maintenance. If I have to reboot everything starts up automatically, image upgrades are a breeze, and I really have no issues with it.

Sure it doesn't have all the bells and whistles of a k8s cluster, but it's perfectly fine for personal use.

I'm still partly annoyed that Swarm is mostly dead in this space and k8s has undoubtedly "won". It's only a matter of time before Docker Inc. fully abandons it. Such a shame.

Agreed, while i've dabbled in Kubernetes, most of my homelab and hybrid cloud infrastructure (think ~10-20 nodes) runs in Docker Swarm with Portainer for graphical administration.

A while back i also tried Nomad, and while it felt polished, it was also a bit lacking in features (manual encryption config, mostly read-only UI, though that was a while back).

It's a shame that Docker Swarm never got more popular, because in my mind it's basically the sweet spot between running containers directly or using Docker Compose and full blown Kubernetes distributions. The Compose manifest format that it uses is really great and clear in my mind, instead of Kubernetes with all of its selectors, object types and so on.

You can use Flux and it will just poll your source control and image registries to see if any updates to the images or configs are available and reconcile automatically. No need to even restart pods. Totally commodity continuous deployment. It's wonderful.

A friend and I have recently stood up homelabs for funsies. Mine's a cluster of x64 boxes, his is a small pile of raspi4s. He went for a k8s cluster off the bat--having worked with cloud stuff for a while, he had the familiarity to hit the ground running.

I was immediately overcome by all the abstractions. I had no idea where to look to figure things out, and was mostly relying on advice from my friend. I dind't know what an ingress controller was, much less how to configure one--but I knew I wanted each 'service' to have its own IP I could route to from my network.

Overall it felt like I had SO MUCH TO LEARN at *every point* it was difficult to get even close to my actual goals (CI/CD for a personal site + some hosted game servers).

I eventually went with the philosophy of "build something now, and move towards perfection later" / "don't let best be the enemy of good" and begain spooling up LXCs and VMs to do the work I needed, planning to move things into k8s later when I better understood the actual things I wanted to move.

(Plus then I got some satisfaction out of actually accomplishing the goals I wanted, instead of just banging my head on k8s documentation and learning all the abstractions.)

As an example, I've not used docker in any meaningful capacity. For anything. No idea how to make a docker image. To k8s the CI for my docker site, I needed to know how to :

1. install the dependencies, which requires compiling a plugin for pandoc, which requires installing haskell and cabal. this is expensive, so I'd prefer to get the pre-res set up once... but that doesn't seem to be how docker works? Do I need an image repository? can I use DockerHub? I've seen HN talk about how docker is trying to monetize, should I run my own repository? Can I do that on my cluster? I'll need an ingress controller to route to it... I don't even know what that is. 2. I need some way to pass the built website files to the contianer I actually want to host them. I think that means I need an NFS share of somekind to store the files, so one coantainer can load them and another can read them. Do I hos tthat on my NAS? I could put an NFS share in the cluster, maybe. No idea how to get Docker to mount one, or k8s to host one. All the examples I seem to find deal more with connecting to remote services on a host than mounting local storage. is it even local storage? 3. Everyone says infrastructure as code is good, so I guess I'll follow this flux tutorial--only to find out the one I followed is out of date, and I should follow their NEW one. But they still assume I know way more baout k8s than I actually do, but still, I'll spend the few hours to get this operational, so then in theory everything else I deploy can be IaC'd, which is just good practice.

At this point I'm so many layers of abstraction deep, I have no idea what I'm actually doing or how concepts relate to each other, and I'm no closer to actually having my goal.

So last night I spent 2 hours spinning up VMs on my cluster and installing dependencies, configuring an nginx proxy, and now I actually have my personal blog self-hosted and updatable. Way more "progress" than the 10ish hours I've sunk into building a k8s cluster already.

There's something to be said for limiting the number of abstractions you're dealing with.

> I didn't know what an ingress controller was

I've seen this comment a lot when discussion introductions to Kubernetes, and it is probably one of the biggest "first step" problems I see, and a perfectly legitimate complaint. If there was one thing that Kubernetes could address to help "onboarding", it might be thing.

Kubernetes is essentially a collection of controllers, that each control a different aspect of the system. These are all "internal", and you don't need to know/understand them in order to stand up Kubernetes in the happy-path. If you're using a managed solution, these are all managed by your provider anyways.

The ingress controllers are the exception. It brings the concept of "controller" out of the "Kubernetes administration" space, and into the "Kubernetes user" space. It opens up a whole can of worms around "What exactly is a controller?" that you shouldn't have to understand in order to get started using Kubernetes.

Part of the issue, I suspect, is I was blending k8s use and k8s administration by running my own k8s cluster on my own equipment--that's likely an extra layer of complexity.

(of course, the whole point of the homelab endeavor I set out on was to be fully self-hosted to learn all these underlying concetps--but definitely doesn't help that I layered some extra abstractions on top of the soup that k8s already is.)

I don't really know anything about Kubernetes either. I've been using Ansible myself for my projects, though they're all single servers. I get the idea that Kubernetes uses Docker heavily, and that you don't understand Docker that well. I'd suggest, if you care to stick to getting into that, learning Docker first.

I have been using Docker for work for a while, though I still don't feel much desire to use it for any personal projects. It seems straightforward enough - basic idea is that each Docker image runs one and only one process, and has it's own directory tree. So if you want to run a Python or Ruby webapp, all of the app files plus the interpreter and all required debs/packages/gems live in that Docker image, and you don't need to have anything else special on your host. A Dockerfile is then a sort of script that holds all of the instructions for setting up an environment your application can run in from scratch.

Of course that means if you want to run a database or a cache server or something, then you'll need to run several Docker images and coordinate communications between them and launching and scaling. I gather that's where Kubernetes and Docker Swarm and other such things come in.

> you don't understand Docker that well

I wholeheartedly agree :)

The path I'm currently taking is to spin up Concourse CI which uses containers, and start using that to CI/CD my personal site. I'm likely gonna overkill and end up hosting my own image repository, but overall I think this'll teach me the necessary concepts of containers in a directly applicable way to my end goals.

From there I can start to play with k8s itself, if I so desire.

Where I tend to get hung up is things like entrypoints. My personal site is static, so the container to build it isn't a hosted service, it's just invoking Pandoc (with a bunch of dependencies installed as well). I think that makes Pandoc the 'entry point', that just seems strange since it isn't a persistent service, and I think that makes the lifetime of the container rather ephemeral.

Which... likely is a valid usecase for docker containers, I've just only interacted with them e.g. on my Unraid box as persistent services.

I gather the actual site is a static files site, and Pandoc is the tool that translates whatever format you're writing them in to the final HTML. So then your site docker image would just contain a server, Nginx or something, and a directory of all of the files for your site, and the correct config for Nginx or whatever to serve them.

The Dockerfile image build process would take care of building that. It's pretty standard for your single project Dockerfile to first assemble an image with a bunch of build tools, like this Pandoc and whatever other dependencies are needed, copy the source files into that image, run the commands to generate the final site, then assemble a second image with just Nginx and copy the output files from the first image onto that to become the final output image. That way, the whole build process is scripted and automated and doesn't require any of the tools on the local system (handy if you were doing stuff like onboarding new contributors), and the final image is small and secure because it only contains a web server and the final HTML pages and no build tools or source files.

You've listed a dozen or so other technologies you've had to learn in the past, which in aggregate required much more effort and experience to master. Much of which provides a good foundation for understanding the concepts that kubernetes builds upon.

I feel like this was more of a hyperbolic rant.

> I feel like this was more of a hyberbolic rant.

Reasonable--it definitely is a bit of a rant. I'm not sure it's hyperbolic, personally, but clearly I have a bias.

Overall, my acute frustration with k8s coming from a native-destkop-development background is I simply don't have the scaffolding to be effective relatively quickly. The learning curve is steep enough that I get discouraged before I actually begin making progress on my own goals, and I just feel like I'm trying to do things The Right Way, without understanding what I'm doing, or why, or actually accomplishing my original goals.

As you said, this is true of concepts I've had to learn in the past, but I could learn all those concepts in isolation, then apply them together. K8s I feel like I have to understand a much larger chunk of before I hit critical mass and can start being effective.

i.e. python, I can start with, like, sqlite, before I move to an external hosted DB. K8s I feel like I have to understand waaaay more components--and they don't directly build. Like Docker-compose _seems like_ a stepping stone to k8s, but I've been told is a false path.

From overview vantage point, YAML-files seems to be stepping stone to even better abstractions. Operators maybe? The orchestration is complex work in itself, and to build up to more advanced setups you need incremental work, or simplifications. In orgs K8s tends to require a separate team.