I'm not sure "a plain ol' Django monolith" with none of the "fancy stuff" is either what people are referring to when they say "kubernetes", or a great choice for that. I could run hello world on a Cray but that doesn't mean I can say I do supercomputing. Our team does use it for all the fancy stuff, and spends all day everyday for years now yamling, terraforming, salting, etc so theoretically our setup is "entire application stack as YAML files, fully reproducible", but if it fell apart tomorrow, I'd run for the hills. Basically, I think you're selling it from a position which doesn't use it to any degree which gives sufficient experience required to give in-depth assessment of it. You're selling me a Cray based on your helloworld.

Reading this charitably: I guess I agree. k8s is definitely overpowered for my needs. And I'm almost certain my blog or my business will never need that full power. Fully aware of that.

But I'm not sure one can find something of "the right power" that has the same support from cloud providers, the open source community, the critical mass, etc. [1]

Eventually, a standard "simplified" abstraction over k8s will emerge. Many already exist, but they're all over the place. And some are vendor specific (Google Cloud Run is basically just running k8s for you). Then if you need the power, you can eject. Something like Create React App, but by Kubernetes. Create Kubernetes App.

[1] Though Nomad looks promising.

Curious why run it at all? The cost must be 10 times more this way. It is mostly for the fun of learning.

I come from the opposite approach. I have 4 servers two digital ocean $5 and two vulr $2.50 instances. One holds the db. One server as the frontend/code. One server to do heavy work and another to server a heavy site and holds backups. For $15 I'm hosting hundreds of sites, running so many background processes. I couldn't imagine hitting that point where k8s would make sense just for myself unless for fun.

Sounds like your setup lacks high availability. If you don't believe you need that, then yeah, kubernetes is overkill.

Few people actually need high availability.

If you do, the recipe is to reduce the number of components, get the most reliable components you can find, and make the single points of failure redundant.

Saying you can use Kubernetes to turn whatever stupid crap people tend to deploy with it highly available, is like saying you can make an airliner reliable by installing some sort of super fancy electronic box inside. You don't get more reliability by adding more components.

> Saying you can use Kubernetes to turn whatever stupid crap people tend to deploy with it highly available, is like saying you can make an airliner reliable by installing some sort of super fancy electronic box inside. You don't get more reliability by adding more components.

This is a bit funny, considering Airbus jets use triple-redundancy and a voting system for some of their critical components. [1]

[1] https://criticaluncertainties.com/2009/06/20/airbus-voting-l...

What about application upgrades?

Are you ok with your application going down for each upgrade? With Kubernetes, it's very simple to configure a deployment so that downtime doesn't happen.

If and only if your application supports it. Database schema upgrades can be tricky for instance, if you care about correctness.

On the other hand, atomic upgrades by stopping the old service and then starting the new service on a Linux command line (/Gitlab runner) can be done in 10 seconds (depending on the service of course – dynamic languages/frameworks sometimes are disadvantaged here). I doubt many customers will notice 10 second downtimes.

And that downtime can even be avoided without resorting to k8s. A simple blue-green deployment (supported by DNS or load balancer) is often all that's needed.

K8s only makes sense at near Google-scale, where you have a team dedicated to managing that infrastructure layer (on top of the folks managing the rest of the infrastructure). For almost everyone else, it's damaging to use it and introduces so much risk. Either your team learns k8s inside out (so a big chunk of their work becomes about managing k8s) or they cross their fingers and trust the black box (and when it fails, panic).

The most effective teams I've worked on have been the ones where the software engineers understand each layer of the stack (even if they have specialist areas of focus). That's not possible at FAANG scale, which is why the k8s abstraction makes sense there.

Takes a couple of minutes at most for an average application upgrade / deployment, a lot of places can deal with that. Reddit is less reliable than what I used to manage as a one man team.

So if a deployment is 2 minutes of downtime, you are limited to 2 per month if you still want to hit 4 9s of availability with no unexpected outages.

You can get a k8s cluster on DO for around $15 p/m. And that itself can host all your apps.

How do you do automated deployments, though? I don't like using K8s for small stuff, but I am also extremely allergic to having to log on to a server to do anything. Dokku hits the sweet spot for me, but at work I would probably use Nomad instead.

Set your pod to pull image always and have entrypoint shell script that clones the repo, kill the pod so on restart you could get your code deployed.

You could run init container with Kaniko that pushes image to repo and then main container that pulls that back but for that you need to do kubectl rollout restart deploy <name>

If you are looking for pure CI/CD gitlab has awesome support or you could do Tekton or Argo. They can run on the same cluster.

What's wrong with logging in to a server? I love logging in to a server and tinkering with it. Sure, for those who operate fleets of hundreds it's not scalable, but for a few servers that's a pleasure.

The problem is when you need to duplicate that server or restore it due to some error, you have no idea what all the changes you made are.

Besides, it's additional hassle and a chance for things to go wrong, the way I have it set up now is that production gets a new deployment whenever something gets pushed to master and I don't have to do anything else.

But this is a solved problem since... well, at least since the beginning of internet. I managed 1000s of Linux & BSD systems over the past 25 years and I have scripts that do all that since the mid 90s that automate everything. I never install anything manual; if I have to do something new, I first write + test a script to do that remotely. Also, all this 'containerization' is not new; I have been using debootstrap/chroot since around that time as well. I run cleanly separated multiple versions of legacy (it is a bit scary how much time it takes to move something written early 2000s to a modern Linux version) + modern applications without any (hosting/reproducibility) issues since forever (in internet years anyway).

That's great but then you're not doing what the commenter above said and "logging in to a server and tinkering with it"

True; I learned many years ago that that is not a good plan. Although, I too, love it. But I use my X220 and Openpandora at home to satisfy that need. Those setups I could not reproduce if you paid me.

> The problem is when you need to duplicate that server or restore it due to some error, you have no idea what all the changes you made are.

A text file with some setup notes is enough for simple needs, or something like Ansible if its more complex. A lot of web apps aren't much more than some files, a database, and maybe a config file or three (all of which should be versioned and backed up).

I would be a lot more confident trying to back up my old school apps than the monstrosity we have on kubernetes we have at present.

Make backup of /etc and package list. Usually that's enough to quickly replicate a configuration. It's not like servers are crashing every day. I'm managing few servers for a last 5 years and I don't remember a single crash, they just work.

I'm logging into a server because i need to, not because its 'pleasure'.

I don't hate it but if you need to login to a server regularly because you need to do an apt upgrade, you should have enabled automatic security updates and not login every few days.

If your server runs full because of some logfiles or stuff, you should fix the underlying issue and not needing to login to a server.

You should trust your machines, independently if it is only one machine, 2, 3 or 100. You wanna be able to go on holiday and know your systems are stable, secure and doing their job.

And logging in also implies a snow flake. Doesn't matter as long as that machine runs and as long as you have not that many changes but k8s actually makes it very simple to finally have an abstraction layer for infrastructure.

Devs went on holiday before k8's.

Yes true, not sure what point you are trying to make.

Flux or Argo can help with this. The operator lives on your cluster, and ensures your cluster state matches a Git repo with all your configuration in it.

Flux - https://github.com/fluxcd/flux

ArgoCD - https://argoproj.github.io/argo-cd/

Write a script which runs remotely over SSH and trigger on the appropriate event in your CI/CD host.

This is what I like to do. In my case, even the CI/CD host is just a systemd service I wrote.

The service just runs a script that uses netcat to listen on a special port that I also configured GitHub to send webhooks to, and processes the hook/deploys if appropriate.

Then when it's done, systemd restarts the script (it is set to always restart) and we're locked and loaded again. It's about 15 lines of shell script in total.

That's quite elegant.

Do you manage to aggregate all logs on a single place? Do you have the same environment as staging? How do you upgrade your servers? Do you have multiple teams deploying on their own components? Do you have a monitoring/metric service? How do you query the results of a Cron? Can you rollback to the correct version when you detect an error at production?

remote syslog has been a thing for how many years?! As has using a standard distribution for your app with a non-root-user for each app, easily wiped and set up every deploy (hint: that's good for security too!). Monitoring was also a solved problem and I guess Cron logs to syslog. Rollback works just like a regular deploy? (I wonder how good k8s helps you with db-schema rollbacks?)

Setting all of that from scratch is not really that easy, and I wouldn't consider "monitoring" to have been a "solved problem". syslog over TCP/UDP had many issues which is why log file shippers happened, and you still need to aggregate and analyze it. Getting application to reliably log remotely is IMO easier with k8s than remote-writing syslog, as I can just whack the developer again and again till it logs to stdout/stderr then easily wrap it however I want.

Deploying as distribution package tends to not work well when you want to deploy it more than once on a specific server (which quickly leads us to classic end-result of that approach, which is VM per deployment minimum - been there, done that, still have scars).

Management of cron jobs was a shitshow, is a shitshow, and probably will be a shitshow except for those that run their crons using non-cron tools (which includes k8s).

Yes, k8s makes it easier and more consistent. But it's not like all the stuff from the past suddenly stopped working or was not possible like GP made it sound ;)

Are you using a framework (cPanel etc) for this or just individual servers talking to each other? Need to move my hosting to something more reliable and cheaper...

I'm learning Elixir now, and it's quite confusing to me how one would go about deploying Elixir with K8s. How much you should just let the runtime handle.

How much of K8s is just an ad hoc, informally-specified, bug-ridden, slow implementation of half of Erlang.

> But I'm not sure one can find something of "the right power" that has the same support from cloud providers, the open source community, the critical mass, etc.

I totally agree. I would dearly like something simpler than Kubernetes. But there isnt a managed Nomad service, and apparently nothing in between Dokku and managed Kubernetes either.

I've been very pleased with Nomad. It strikes a good balance between complexity and the feature set. We use it in production for a medium sized cluster and the migration has been relatively painless. The nomad agent itself is a single binary that bootstraps a cluster using Raft consensus.

I was about to, but seems like you answered the question yourself through that footnote.

>but if it fell apart tomorrow, I'd run for the hills

The test is going from zero to production traffic in a new cloud region.

So are you saying that, no matter what, if you want to reply your whole infrastructure as code (networks, dmz, hosts, services, apps, backups etc ) that you are going to have to reproduce that somehow (whatever the combo of AWS services are, OR just learn K8S

Effectively, "every infrastructure as code project will reimplement Kubernetes in Bash"

Not necessarily. You can have all of above with Terraform, Ansible, Puppet, Chef... etc.

Instead of a varied interface tools set, I can have one with consistent interfaces and experiences. Kubernetes is where everything is going, hence why TF and Ansible have both recently released Kubernetes related products / features. It's their last attempt at remaining relevant (which is more than likely wasted effort in the long run). They have too much baggage (existing users in another paradigm) to make a successful pivot.

Ironically for me, those two tools are part of the blessed triad that we use for all of our infrastructure as code and end-user virtual machine initial setup.

If we only got to keep two tools it would be kubernetes and terraform.

Last attempt at remaining relevant - lol.

This isn’t a competition, they are tools. Ansible is widely used and will continue to be so for a long long time. Its foundations - ssh, python and yaml are also in for the long run to manage infrastructure...

Yaml is on the way out, Cuelang will replace it where it's used for infra. It's quite easy to start by validating yaml and then you quickly realize how awesome having your config in a well thought-out language is!

I thought you were trolling with some called Cuelang but is actually a thing.

Yaml will still be used in 100 years, k8s is yaml based...

Yes but YOU won't be writing or seeing yaml, Cue will output the yaml and run kubectl (I'm already doing this to great relief)

Which is step 2 of my Enterprise adoption strategy. Step 1 is starting with validation, step 3 and beyond is where the real fun starts!

Hope that works for you ! Your statement above has nothing to do with yaml being on the way out...

well, then perhaps the growing frustration with a configuration language where meaning depends on invisible chars is an argument. And then what helm and others are doing text interpolating and add helpers for managing indentation.

There are many experiments into alternatives happening right now, so I do believe yaml's days are numbered. I'm actively replacing it where ever I encounter it with a far superior alternative. Cue is far more than a configuration language however, worth the time to learn and adopt at this point.

There are so many configuration files format and language already. The problem you describe seems to be fixed by XML for example

Exactly. That's a great way to put it. A bunch of Bash glue reimplementing what Kubernetes already does. Poorly.

Kubernetes is outstanding because proving ol bash scripts were bad and re-writing all Bash glue in Go glue, slapping 150 APIs on top (60 "new" and about hundred "old versions" https://kubernetes.io/docs/reference/generated/kubernetes-ap...), adding few dozen opensource must have projects - so finally it can be called "cloud native" - boom, a new cloud native bash for the cloud is born!

Actually it's worse than that - k8s was ported from Java, and the codebase is an incoherent mess, as k8s devs say themselves

Bash glue has rarely failed me. Never used k8s, but the horror stories I hear ensure I won’t feel dirty for writing a little bash to solve problems anytime soon.

Swarm is still supported and works. I have it running on my home server and love it.

Kubernetes is fine, but setting it up kind of feels like I'm trying to earn a PhD thesis. Swarm is dog-simple to get working and I've really had no issues in the three years that I've been running it.

The configs aren't as elaborate or as modular as Kubernetes, and that's a blessing as well as a curse; it's easy to set up and administer, but you have less control. Still, for small-to-mid-sized systems, I would still recommend Swarm.

> setting it up kind of feels like I'm trying to earn a PhD thesis.

The kind of people who has to both set the cluster up and keep it up and also has to develop the application and deploy it and keep it up etc is not the target audience.

K8s shines when the roles of managing the cluster and running workloads on it are separated. It defines a good contract between infrastructure and workload. It lets different people focus on different aspects.

Yes it still has rough edges, things that are either not there yet, or vestigial complexity of wrong turns that happened through it's history. But if you look at it through the lense of this corporate scenario it starts making more sense than when you just think of what a full-stack dev in a two person startup would rather use and fully own/understand.

One of the things nobody liked to talk about in public when test automation was slowly "replacing" testers is that if you had the testers write automation, they brought none of the engineering discipline we tend to take as a given to the problem.

It's hard to make tests maintainable. Doubly so if you aren't already versed in techniques to make code maintainable.

I wonder sometimes if we aren't repeating the same experiment with ops right now.

There are elements of our company that want to move to Kubernetes for no real reason other than it's Kubernetes. I can't wait to see the look on their faces when they realise we'll have to employ someone full-time to manage our stack.

I'm not sure you have to, that seems like the whole point of managed services like GKE.

Do you have a recommended tutorial for engineer with backend background to setup a simple k8 infra in ec2, I am interested in understanding devops role better

What are Kubernetes' rough edges?

> setting it up kind of feels like I'm trying to earn a PhD thesis.

Are you following "k8s the hard way"? I've never had this problem; either:

`gcloud container clusters create`

Or

`install docker-for-mac`

And you have a k8s cluster up and running. Maybe it's more work on AWS?

Ekscli (if you want managed) and kops (if you want to manage it yourself) are just as straightforward.

There's an offshoot of this that I see from developers, especially at old, stodgy companies.

Once everything is "infrastructure as code", the app team becomes less dependent on other teams in the org.

People like to own their own destiny. Of course, that also removes a lot of potential scapegoats, so you now mostly own all outages, tech debt, etc.

I think that's been coming since ... well ever.

I worked in networking for the longest time. When I started there network guys and server guys (at least where I was). They were different people who did different things who kinda worked together.

Then there were storage area networks and similar, networks really FOR the server and storage guys.... that kind of extended the server world over some of the network.

Then comes VMware and such things and now there was a network in a box somewhere that was entirely the server guy's deal (well except when we had to help them... always).

Then we also had load balances who in their own way were a sort of code for networks ... depending on how you looked at it (open ticket #11111 of 'please stop hard coding ip addresses').

You also had a lot of software defined networking type things and so forth brewing up in dozens of different ways.

Granted these descriptions are not exact, there were ebs and flows and some tech that sort of did this (or tired) all along. It all starts to evolve slowly into one entity.

Conversely I think the trend of writing infrastructure as yaml may be the worst part of modern ops. It’s really hard to think of a worse language for this.

Fortunately YAML is not actually necessary in k8s, and only provided as a convenience because writing JSON by hand (or Proto3, lol) is annoying verging on insane.

We can build higher level abstractions easily having a schema to target and we can build them in whatever we want. That's a big boon for me :)

For kubernetes thats absolutely true and I think more people should do that (disclaimer I work with this https://github.com/stripe/skycfg daily). I think it actually would be easier for people to understand the k8s system if they did.

But, yaml is now everywhere in the ops space. Config management systems use it, metrics systems use it, its the defacto configuration format right now and that is unfortunate cause its bad.

Why?

We have plenty of yaml 'code' which is simple and does exactly what it needs to do.

For all other usecases, there are plenty of alternatives, including libs for your preferred language.

Most people use the yaml way because its easy and does exactly what it needs to do.

Everyone else has plenty of well supported and well working alternatives.

Its typing is too weak. It’s block scoping is dangerous. References, classes and attachments are all pretty bad for reuse. Schemas are bolt on and there is no standard query language for it.

> I can and have repointed my entire infrastructure with minimal fuss.

When you get to that blog post please consider going in depth on this. Would love to see actual battletested information vs. the usual handwavy "it works everywhere".

I sure will. 99% of the work was ingress handling and SSL cert generation. Everything else was fairly seamless.

Even ingress is trivial if you use a cloud balancer per ingress. But I wanted to save money so use a single cloud balancer for multiple ingresses. So you need something like ingress-nginx, which has a few vendor-specific subtleties.

Have you tried or considered Nomad (from the makers of Terraform)?

I haven't, and since I've sunk the cost into Kubernetes and know it very well now, likely won't end up there.

In retrospect though, maybe it's exactly what I needed. Great suggestion.

I've been using Nomad for my "toy" network, and I like it. It runs many services, and a few periodic jobs. Lightweight, easy to set up, and has enough depth to handle some of the weirder stuff.

Nomad, in its free offering, cannot compete with k8s for organization-wide usage:

- no RBAC

- no quotas

- no preemption

- no namespacing

This means: everyone is root on the cluster, including any CI/CD system that wants to test/update code. And there's no way to contain runaway processes with quotas/preemption.

can it handle networking (including load balancing and reverse proxies with automatic TLS) or virtualized persistent storage? Make it easy to integrate common logging system?

Cause those are the parts that I miss probably the most when dealing with non-k8s deployment, and I haven't had the occasion to use Nomad.

For load balancing you can just run one of the common LB solutions (nginx, haproxy, Traefik) and pick up the services from the Consul service catalog. Traefik makes it quite nice since it integrates with LetsEncrypt and you can setup the routing with tags in your Nomad jobs: https://learn.hashicorp.com/nomad/load-balancing/traefik

What Nomad doesn’t do is setup a cloud provider load balancer for you.

For persistent storage, Nomad uses CSI which is the same technology K8s does: https://learn.hashicorp.com/nomad/stateful-workloads/csi-vol...

Logging should be very similar to K8S. Both Nomad and K8S log to a file and a logging agent tails and ships the logs.

Disclosure, I am a HashiCorp employee.

Thanks, definitely widened my understanding of Nomad in pretty short time :)

Kinda feels bad that I don't have anything to use it on right now.

Does the Nomad WebUI support any kind of auth or just the Nomad-Bearer thing?

Thinking about completing my Hashicorp Bingo card.

It is on the roadmap to support JWT/OIDC!

Those are the advantage and the problem of nomad. We're using it a lot by now.

Nomad, or rather, a Nomad/Consul/Vault stack doesn't have these things included. You need to go and pick a consul-aware loadbalancer like traefik, figure out a CSI volume provider or a consul-aware database clustering like postgres with patroni, think about logging sidecars or logging instances on container hosts. Lots of fiddly, fiddly things to figure out from an operative perspective until you have a platform your development can just use. Certainly less of an out-of-the-box experience than K8.

However, I would like to mention that K8 can be an evil half-truth. "Just self-hosting a K8 cluster" basically means doing all of the shit above, except its "just self-hosting k8". Nomad allows you to delay certain choices and implementations, or glue together existing infrastructure.

K8 requires you do redo everything, pretty much.

I count the "glue it with existing infrastructure" to be higher cost than doing it from scratch. It was one feature that I definitely knew regarding Nomad, as one or two people who used it did chime in years ago in discussion, but for various reasons that might not be applicable to everyone I consider those unnecessary complication :)

Depending on how big the infrastructure is and how long you want to migrate over... usually there's not enough resources to "redo all from scratch", millions of LoC are already in production, people who owned key services are no longer in the company, other priorities for business exists other than have what you have already working in k8s..

The context was rather different (home setup), but all that you mention can be used as arguments Noth for and against redo, basing on situation in company, future needs, etc.

I have actually done a "lift and shift" where we moved code that had no support or directly antagonistic one to k8s because various problems reached situation where CEO said "replace the old vendor completely" - we ended up using k8s to wrestle with the amount of code to redeploy.

- Yes (Traefik, Fabio, consul connect/envoy)

- Yes, just added CSI plugin support. Previously had ephemeral_disk and host_volume configuration options, as well as the ability to use docker storage plugins (portworx)

- I haven’t personally played with it, but apparently nomad does export some metrics, and they’re working on making it better

nomad is strictly a job scheduler. If you want networking you add consul to it and they integrate nicely. Logging is handled similarly to Kubernetes. Cool thing about Nomad that it's less prescriptive

With "Swarm", do you mean Docker Swarm? Why has it "fizzled"?

The way I learned it in Bret Fisher's Udemy course, Swarm is very much relevant, and will be supported indefinitely. It seems to be a much simpler version of Kubernetes. It has both composition in YAML files (i.e. all your containers together) and the distribution over nodes. What else do you need before you hit corporation-scale requirements?

I use Swarm in production and am learning k8s as fast as possible because of how bad Swarm is:

1. Swarm is dead in the water. No big releases/development afaik recently

2. Swarm for me has been a disaster because after a couple of days some of my nodes slowly start failing (although they’re perfectly normal) and I have to manually remove each node from the swarm, join them, and start everything up again. I think this might be because of some WireGuard incompatibility, but the strange thing is that it works for a week sometimes and other times just a few hours

3. Lack of GPU support

To add another side, I use Swarm in production and continue to do so because of how good it is.

I've had clusters running for years without issue. I've even used it for packaging B2B software, where customers use it both in cloud and on-prem - no issues whatsoever.

I've looked at k8s a few times, but it's vastly more complex than Swarm (which is basically Docker Compose with cluster support), and would add nothing for my use case.

I'm sure a lot of people need the functionality that k8s brings, but I'm also sure that many would be better suited to Swarm.

Yeah I guess for smaller projects and the addition of using Docker Compose files, Swarm would be worth it.

If K8s supported compose scripts out of the box (not Kompose) that'd basically make Swarm unnecessary (at least for me)

This happened to me too when I was using swarm in 2017. Had to debug swarm networks where nodes could send packets one way but not the other. Similar problems as #2 where stuff just breaks and resetting the node is the quickest way I found to fix it.

Switched to k8s in late 2017 and it’s been much more solid. And that’s where the world has moved, so I’m not sure why you’d choose swarm anymore.

> What else do you need before you hit corporation-scale requirements?

Cronjobs, configmaps, and dynamically allocated persistent volumes have been big ones for our small corporation. Access control also, but I'm less aware of the details here, other than that our ops is happier to hand out credentials with limited access, which was somehow much more difficult with swarm

Swarm has frankly also been buggy. "Dead" but still running containers - sometimes visible to swarm, sometimes only the local Docker daemon - happen every 1-2 months, and it takes forever to figure out what's going on each time.

A little off topic but why do these orchestration tools always prefer to use YAML as I really feel it a harder to understand format than JSON or better TOML and it's the only thing that I don't like about Ansible.

I see ruby kind of uses YAML often but are people comfortable editing YAML files? I always have to look up how to do arrays and such when I edit them once in a while.

> when I edit them once in a while

Anything new needs a certain amount of sustained practice before you get the hang of it. I think I had to learn regex like four times before it stuck. I haven’t hit that point with TOML yet so I avoid it.

I’d suggest using the deeper indentation style where hyphens for arrays are also indented two spaces under the parent element. Like anything use a linter that enforces unambiguous indentation.

I prefer YAML for human-writeable config because JSON is just more typing and more finicky. The auto-typing of numbers and booleans in YAML is a pretty damn sharp edge though and I wish they’d solved that some other way.

Yup, even monoliths can benefit from certain k8s tooling (HPAs, batch jobs, etc).

"What cron jobs does this app run?"

Open cron.yaml and see. With schedule. Self documented.

Amazing. Every time. Even when some as my k8s battle wounds are still healing (or permanently scarred). See other replies for more info.

Just like my Ansible repo.

Do you have a recommended tutorial for engineer with backend background to setup a simple k8 infra in ec2?

Take a look at https://github.com/kelseyhightower/kubernetes-the-hard-way.

That’d be a great first step if the purpose is to learn Kubernetes. If, however, you want to set up a cluster for real use then you will need much more than bare bones Kubernetes (something that solves networking, monitoring, logging, security, backups and more) so consider using a distribution or a managed cloud service instead.

Setting up your own k8s from scratch is kind of like writing your own string class in C++: it’s a good exercise (if it’s valuable for your learning path) but you probably don’t want to use it for actual work.

Maintaining a cluster set up like that is a ton of work. And if you don’t perform an upgrade perfectly, you’ll have downtime. Tools like kops help a lot but you’ll still spend far more time than the $70/month it costs for a managed cluster.

I find that K3s is great for getting started. It has traefik included and its less of a learning curve to actually be productive, vs diving in with K8s and having to figure out way more pieces.

I've just started to look into it, but it seems like the project has been focusing on improving the onboarding experience since it has a reputation for being a huge pain to set up. Do you think it has gotten easier lately?

No. Not easier in my opinion. And some of the fires you only learn after getting burnt badly. [1]

Note: my experience was all with cloud-provided Kubernetes, never running my own. So it was already an order of magnitude easier. Can't even imagine rolling my own. [2]

[1] My personal favorite. Truly egregious, despite how amazing k8s is. https://github.com/kubernetes/kubernetes/issues/63371#issuec...

[2] https://github.com/kelseyhightower/kubernetes-the-hard-way

Rancher makes this quite painless; worth a look if you have to run on prem, or in an iaas cloud for some reason.

Out of curiosity why do you feel swarm fizzled out?

I’ve deployed swarm in a home lab and found it really simple to work with, and enjoyable to use. I haven’t tried k8, but I often see view points like yours stating that k8 is vastly superior.

Of course it doesn't have the advantage of #2, but I've found ECS to be far easier to grok and implement.

According to the article you are wrong about "infrastructure as code". Kubernetes is infrastructure as data, specifically YAML files. Puppet and Chef are infrastructure as code.

Edit: not sure why the down votes, I was just trying to point out what seems like a big distinction that the article is trying to make.

Maybe? I'm too lazy to formally verify if the YAML files k8s accepts are Turing complete. With kustomize they might very well be.

How about "infrastructure-as-some-sort-of-text-file-versioned-in-my-repository". It's a mouthful, but maybe it'll catch on.

They don't do loops or recursion. They don't even do iterative steps in the way that Ansible YAML has plays/tasks.

Yes, higher-level tools like Kustomize or Jsonnet or whatever else you use for templating the files are Turing-complete - but that's at the level of you on your machine generating input to Kubernetes, not at the level of Kubernetes itself. That's a valuable distinction - it means you can't have a Kubernetes manifest get halfway through and fail the way that you can have an Ansible playbook get halfway through and fail; there's no "halfway." If something fails halfway through your Jsonnet, it fails in template expansion without actually doing anything to your infrastructure.

(You can, of course, have it run out of resources or hit quota issues partway through deploying some manifest, but there's no ordering constraint - it won't refuse to run the "rest" of the "steps" because an "earlier step" failed, there's no such thing. You can address the issue, and Kubernetes will resume trying to shape reality to match your manifest just as if some hardware failed at runtime and you were recovering, or whatever.)

Infrastructure-as-config?

What’s the material difference between well-formatted data and a DSL? Why does this matter?

Weird that you're getting downvoted.

The difference between code and data is pretty big.

One implies an expectation that the user is going to write some kind of algorithm whereas the other is basically a config file.

I think you could think of "infrastructure as code" as he described it as a superset of "infrastructure as data". Both have the benefit of being able to be reproducibly checked into a repo. Declarative systems like Kubernetes/"infrastructure as data" just go even further in de-emphasizing the state of the servers and make it harder to get yourself into unreproducible situations.

Seems like a nitpick? Infra as data seems like a subset of infrastructure as code

Do you have a good tutorial for doing Django or a standard 3 tier web app on Kubernetes? We are using kubernetes at my workspace, but it seems way too complicated to consider for something like that. Maybe if I can bridge the gap between architectures it will help.

Out of curiosity, are you using terraform to deploy k8s, your app stack on k8s, or both?

To follow this - what do you feel K8S provides on top of terraform?

We used K8S on a large project and I felt like it really, really wasn't necessary.

Nothing. We use Terraform to provision a simple auto-scaling cluster with loadbalancers and certs, does exactly the same thing but there is no Docker and k8s. Few million lines less Go code turning yaml filed into seggfaults.

Consistency and standardized interfaces for AppOps regardless of the hyper-cloud I use. Kubernetes basically has an equivalent learning curve, but you only have to do it once

They operate at different layers. K8s sits on top of the infrastructure which terraform provisions. It's far more dynamic and operates at runtime, compared to terraform which you execute ad-hoc from an imperative tool (and so only makes sense for the low level things that don't change often).

So terraform is a higher-order, meta-Kubernetes. It's very rarely used, but who provisions the cluster itself? That's terraform.

So terraform creates the cluster, DNS and VPC. Then k8s runs pretty much everything.

How are you deploying the workloads into the cluster? Manual kubectl or Helm, GitOps with something like Flux, something else?

Spinnaker. Huge. Clunky. But excellent if you can justify it.

Alas, still rely on bash for that. Practically a one liner.

Mainly just kustomize piped into kube apply.

But, but, but. Having to a create a one-off database migration script imperatively.

Ooh!

Some others are Tekton, Argo, KNative You could also use Jenkins with K8s deploy plugin (from MS Azure Devops team)

Why would you want to provision your own k8s cluster, if you can use EKS, AKS or similar?

I don't. I use a cloud cluster. But that still has to be provisioned? You need to choose size, node pool, VPC, region, etc.

EKS, GKE and the like have a number of limitations. For example: they can be pretty far behind in the version of K8S they support (GKE is at 1.15 currently, EKS at 1.16; K8S 1.18 was released in at the end of March this year.

One example I worked on myself: when you need to train lots of ML models and need lots of video cards. Those would be damn expensive in the cloud!

imo, all swarm needed was a decent way to handle secrets and it would have been the k8s of its day