> Getting Kubernetes up and running isn't really the issue anymore, that's pretty easy to do. The tricky part is long term maintenance and storage.

This times 100. Deploying basic clusters is easy. Keeping a test/dev-cluster running for a while? Sure. Keeping production clusters running (TLS cert TTLs expiring, anyone?), upgrading to new K8s versions, proper monitoring (the whole stack, not just your app or the control-plane), provisioning (local) storage,... is where difficulties lie.

I’m working on this right now. My theory is that having every cluster object defined in git (but with clever use of third party helm charts to reduce maintenance burden) is the way to go.

Our cluster configuration is public[1] and I’m almost done with a blog post going over all the different choices you can make wrt the surrounding monitoring/etc infrastructure on a Kubernetes cluster.

[1] https://github.com/ocf/kubernetes

My comment above did not come out of the blue, but based on real-world experience ;-) You may be interested in our MetalK8s project [1] which seems related to yours.

[1] https://github.com/scality/metalk8s

This looks really cool. Would you include dev apps in the mono repo too? Say you have 40 services by 20 different teams.

If I already have an production cluster up and running for a while, how can I get it git tracked now?

Another question: how about things like autoscaler which automatically edits numbers in deployments, how to git track that?

first of all with k3s keeping a production cluster running is still pretty easy.

second you should always be ready to start from scratch, which is also pretty simple, because of terraform.

a lot of people are scared of k8s but they did not even try. they prefer to maintain their scary ansible/puppet whatever script that works only half as good as k8s.

> first of all with k3s keeping a production cluster running is still pretty easy.

Fair enough. I'll admit I have no direct experience with K3s. There are, however, many K8s deployment systems out there which I would not consider 'production-ready' at all even though they're marketed that way.

> second you should always be ready to start from scratch, which is also pretty simple, because of terraform.

That may all be possible if your environment can be spawned using Terraform (e.g., cloud/VMWare environments and similar). If your deployment targets physical servers in enterprise datacenters where you don't even fully own the OS layer, Terraform won't bring much.

> a lot of people are scared of k8s but they did not even try. they prefer to maintain their scary ansible/puppet whatever script that works only half as good as k8s.

We've been deploying and running K8s as part of our on-premises storage product offering since 2018, so 'scared' and 'didn't try' seems not applicable to my experience. Yes, our solution (MetalK8s, it's open source, PTAL) uses a tech 'half as good' as K8s (SaltStack, not Ansible or Puppet) because you need something to deploy/lifecycle said cluster. Once the basic K8s cluster is up, we run as much as possible 'inside' K8s. But IMO K8s is only a partial replacement for technologies like SaltStack and Ansible, i.e., in environments where you can somehow 'get' a (managed) K8s cluster out of thin air.

I've been using this terraform provider quite a lot lately. It has made it a cinch to templatize a full manifest and pass data to the template for a deploy. We now have a fully reproducible base EKS cluster deploy done with working cert-manager/letsencrypt, nginx ingress, weave-net, fluentd logging to elasticsearch service, etc. Our application-specific code lives in a different repo and deploys things using YTT. It's so much more elegant than our old method of copying and pasting manifests and crossing our fingers and hoping the cluster didn't fall down. A full migration to a new cluster and deploy of a whole new application stack takes under an hour now.

https://github.com/gavinbunney/terraform-provider-kubectl

This is where Uffizzi is going. Phase 1 they started with their own custom controller and they are managing clusters for customers who pay for a portion of that cluster. In Phase 2 they are opening up their control plane to enterprise customers to solve the “times 100” management issue listed above.

I was one of those backwards people who opposed containers, until late 2019.

My start was the official kubernetes docs, step by step. Try everything, write it down, make ansible playbooks. Even used some of their interactive training modules, but I quickly had my own cluster up in vagrant so didn't really need the online shell.

Now we have two clusters at work, I have ansible playbooks I'm really happy with that help me manage both our on-prem clusters and my managed LKE with the same playbooks.

I'm completely sold on this container thing. :)

>Now we have two clusters at work

Running your two tightly coupled distributed monoliths of your front and back office systems no doubt!

Heh no single project, staging and prod. In fact prod isn't ready, that's when I need to start learning HA control plane.

Do the playbooks rely on kubeadm internally for setup?

Are they OSS by any chance? And what kind of tasks do they do?

Yes but I won't link them.

They are focused only on CentOS 7, you might be able to find them yourself. As far as I can tell they include everything except persistent storage and HA control plane.

Just use Kubespray, not much sense in wasting time writing your own unless your intention is specifically learning how to manage cluster setup with stateless Ansible

I actually looked at kubespray first but first of all it did not work out of box as promised. And to troubleshoot it you basically had to know about kubernetes already.

So it made a lot of sense to start at the official docs. And while I'm reading the docs, why not build the cluster at the same time. And while I'm building the cluster, why not write each step down in Ansible so I won't have to repeat myself.

So end result is my own ansible setup that I'm happy with and I know inside and out.

It always bothered me to run other people's ansible playbooks. I'm too much of a control freak.

When people talk about “bare metal” k8s they mean on-prem without the support of integrated cloud infrastructure. All the external stuff like load balancers, ingress controllers, routing, bgp, public ip pools, shared storage, vm creation for scaling, are all things the cluster can’t do for itself and have to be implemented and integrated.

Shared storage being a particular head-f to get solved. Everything else is a reasonably approachable problem you can tackle as it comes.

https://rook.io/ is a really great way to get shared storage up quickly and simply.

I'm still looking for a good resource as someone who's taken their eye off the ball for a decade or so. Things have (and continue to) move at an astounding pace...and I'm needing something that goes from 'Pets and autoconf' to where we are now.

It's still using the same old stack, just containerized. For example, you're no longer manually editing nginx conf, but instead writing the config as yaml annotation to tell the nginx ingress what to do. Instead of editing wp-config.php when deploying wordpress, you specify the db config as environmental variables in yaml file, etc.

Can you explain why one would use Kubernetes? I know containerization but still haven’t figured out wtf Kubernetes is. Everyone and their mother keeps telling me it’s an orchestration tool for containers. It’s as helpful as saying a Pecan pie is an orchestration of flour, sugar and pecans.

Orchestration systems like Kubernetes, Amazon ECS, and (formerly, it’s pretty dead) Mesos run on a cluster of systems and coordinate over the network. What they’re coordinating is running your containerized services, for example your API server, micro services, etc. So you can tell the orchestration service “run 5 copies of this and listen externally on port 9000. When a request comes in on external_ip:9000, redirect it to one of the 5 service instances.” They will also restart your services if they die, keep stdout/stderr logs, and so forth. You can have very complex setups but most will never need more than what I’ve just described.

The main difference between the different orch. tools is how they pipe the data between Outside and the service cluster. Mesos didn’t help you, you had to build your own service discovery. ECS uses Amazon load balancers and some custom black-box daemons. Kubernetes famously uses a complex mesh of “ingress controllers” and kernel iptables/ipfw spoofing. All take their configurations in some form of JSON or YAML.

Thank you.

Is there some sort of a PID-feedback-loop control that monitors the CPU/memory load and helps spin up more instances if it sees more traffic? If Kubernetes doesn't do that, what piece of software can help automatically scale if there is a huge traffic spike?

Load balance AFAIK doesn't do that. It just helps distribute the load.

Using a HorizontalPodAutoscaler [1] you can scale up and down the amount of "pods" (a grouping of containers that are scheduled as a unit) based on the desired metric.

[1]: https://kubernetes.io/docs/tasks/run-application/horizontal-...

Autoscaling is a field of its own, yes. Lots of footguns, though.

Kubernetes is a multi-physical-machine operating system. In a typical OS you have CPU, RAM, etc. and the kernel decides how much of each to allocate to each running task. Kubernetes works the same way, but across multiple computers by default- and it can additionally manage any other arbitrary resource through the use of CustomResourceDefinitions (IPs, higher level concepts like “Minecraft Servers”, etc.)

If you want a (better) cooking analogy - if a container is a 'station' or a line cook in a professional kitchen, then Kubernetes is 'running the line', managing the kitchen, making sure everyone has the tools and ingredients they need.

"Orchestration" means it will manage how many of each container you have running, configuring, starting, and stopping them and possibly migrating them from one machine to another.

The term was created for virtual machines.

if you know systemd, one of its many functions is a single stop systemd controller for all of your infrastructure. Then it also gives you a single way to control storage, monitoring, load balancing and deployment from a single API.

It's neat. It is a lot of moving parts tho. I am just now trying it in a big infrastructure because we have so many bespoke parts that we have to glue together that we might as well try to use what is standard now...

>For example, you're no longer manually editing nginx conf,

You now have to do the configuration in YAML, which is MUCH MUCH worse.

Or more accurately, you now have to understand both the nginx.conf file format, and the right way to nudge the YAML so it produces what you want. Plus some intermediate wrapper script that sets things up on container start that does more funky things based on environment variables.

I don't get it, how do you configure nginx with a YAML file?

You don't directly. You add some extra lines of yaml to an k8s ingress resource, nginx detects this and updates itself. A single nginx container can service many ingresses for each of your apps. The idea is to distribute the config to the app manifests they are related to

> Getting Kubernetes up and running isn't really the issue anymore, that's pretty easy to do.

Care to share an "easy" recipe, because I haven't found one that actually works. It always falls apart for me with the networking.

Let's say I have a cluster of 8 physical nodes and a management node available in my data center and I want to set up k8s for use by my internal users. I'm a solo admin with responsibilty for ~250 physical servers so any ongoing management necessary will be very much a task among many.

Is this blog post a good guide? Is there a better one?

An 'easy' way to deploy a cluster could be using kubeadm. Then you'll need a CNI like Calico to get Pod networking up-and-running. However, you'll want to install a bunch of other software on said cluster to monitor it, manage logs,...

Given you're running on physical infrastructure, MetalK8s [1] could be of interest (full disclosure: I'm one of the leads of said project, which is fully open-source and used as part of our commercial enterprise storage products)

[1] https://github.com/scality/metalk8s