Swarm might be great, but the ecosystem around k8s is massive. So my two choices are: ignore Docker entirely and do it like I've always done (Heroku, chef, puppet, etc. just managing my servers myself), or buy into containers and go with K8S.
To that extent, I only ever deal with K8S at work, and for my personal stuff I don't go near Docker at all. Why would I when I can get an 8GB/4CPU VPS for $9/mo and serve everything I care about from that? I've got fail2ban, systemd, caddy, and plenty of resources.
I can host mysql on the same machine because there isn't a god-given law about hosting your DB on a separate box. Same with anything else.
No need for Docker or Swarm or K8s.
I ignore docker myself. It's been too unstable for me and historically just costs me time, I'd rather just use systemd on a VPS. Occasionally when someone else has setup a complex system in docker I want to play with it has been convenient. Still, I avoid installing it at all.
This is exactly why I think new-cloud companies like fly.io, stackpath, deno.land, vercel.app are the future. No one wants to deal with muck if they can avoid it.
Cloud exposed data center primitives. Then, cloud-native laid bare the at-scale primitives. Imho, I think the industry is ripe for both NoCloud and NewCloud.
Did you read the article? It was about a company struggling to monetize an open source project. It hardly seems time for teenagerish M$-Windows asides on how you don't like people monetizing the software they are giving you for free.
Except that they didn't. With even a little bit of research you can easily find out that they made skipping updates a paid feature, and then even removed that after negative feedback!
People were even talking about downgrading below the version that introduced the 'paid update skipping', so any way you look at it, your comment doesn't make any sense.
- deploying k8s on bare metal is not easy. It was never really designed for this and to be fair Google has a vested interest in making you use GCP. EKS is nothing more than cobbled together scripts and you still need to do a lot of manual work
- k8s introduces a ton of terminology and new concepts (pods/service/deployment/ingress) not to mention the whole host of addons built on top like Istio.
It shouldn't take any decent dev more than a couple days to install minikube/k3s or use a cloud provider, play around and get comfortable with k8s.
- Cert Renewal / Rotation: https://docs.docker.com/engine/reference/commandline/swarm_c...
- TLS Termination: Arguably this is an application layer concern and out of scope for an orchestration layer, but in either case use Nginx (or your favorite TLS terminating proxy) as your ingress point. This is equivalent to the Nginx ingress controller in k8s.
- Storage: Docker volumes (local or ceph, for distributed).
- Disaster Recovery: Node failure? Cluster failure? Data loss? What kind of DR? The strategies you'd use with Swarm will be very similar to those you'd use with k8s.
- Grouping resources: Labels and/or services.
In a Swarm world you'll have fewer moving pieces, far lower resource requirements, and will manage it all in a docker-compose file(s). You'll also get things like service discovery, routing, and secrets management for "free". No additional configuration needed.
You linked Swarm CA renewal operations; this is maybe 1/50 of the functionality of cert-manager. Is there a product like cert-manager that I can run on Swarm that's tightly integrated?
Why not have ingress intrinsically linked to the underlying reaources like k8s does? Your suggestion means that someone needs to come up with a methodology and automation to update ingress with Swarm. Yeah, it's obviously possible but imo, Traefik ingress reaources are way easier to manage with config spread across far fewer places.
Storage.. I have an nfs storage provider in k8s which allows me to dynamically create storage with lifecycle management included... for free... just because I'm using k8s.
DR as in something happens to my house or other technical issues which bring down my cluster. I would posit that it's much easier for me to restore a cluster because it's all yaml and imo more descriptive and capable than swarm yaml syntax.
I think you get my point. Yes, you can do it all with Swarm but with things like k3s and k0s, it's just as easy to start using k8s. And way more help, docs, and tooling to boot.
"I'm going to use this product/project."
"You shouldn't use that, use this instead."
"But this product/project has all these features I want."
"Sure but you can just build and maintain those features yourself if you want. It's open source, after all!"
Or alternately:
"Well no one really needs those anyway, it's just people being told that they do."
I've been hearing this same refrain for over twenty years now. It seems as though some people don't understand that there's actual value in having a system that just works, rather than one that you have to build and maintain yourself just to emulate the system that works.
Specifically, it feels like you are missing that the people who argue against k8s are saying that it is extremely difficult to configure, and so instead of this clean "some people don't understand that it is nice to have a system that just works", it feels like the argument is actually about a tradeoff between what you think is easier to put together: a single system with a million options that seem to all need to be configured or it doesn't work or a handful of systems that are each easy to understand in separate, but where you will need to do all the glue manually.
You're no more right or wrong here than anyone else is.
> It seems as though some people don't understand that there's actual value in having a system that just works, rather than one that you have to build and maintain yourself just to emulate the system that works.
This, however, feels like a lie.
It might "just work" because of the efforts of hundreds of engineers that are in the employ of the larger cloud vendors, because they're handling all of the problems in the background, however that will also cost you. It might also "just work" in on prem deployments because you simply haven't run into any problems yet... and in my experience it's not a matter of "Whether?" but rather one of "When?"
And when these problems inevitably do surface, you have to ask yourself, whether you have the capacity to debug them and mitigate them. In the case of K3s clusters, or another lightweight Kubernetes distro that might be doable. However, if you're dealing with a large distro like RKE or your own custom one, then you might have to debug this complex, fragmented and distributed system to get to the root cause and then figure out how to solve it.
There's definitely a reason why so many DevOps specialists with Kubernetes credentials are so well paid right now. There's also a reason for why many of the enterprise deployments of Kubernetes that i've seen are essentially black holes of manhours, much like implementing Apache Kafka when something like RabbitMQ might have sufficed - sometimes because of chasing after hype and doing CV driven development, other times because the people didn't know enough about the other options out there. There's also a reason for why Docker Swarm is still alive and kicking despite how its maintenance and development have been mismanaged, as well as why HashiCorp Nomad is liked by many.
In short:
The people who will try to build their own complex systems will most likely fail and will build something undocumented, unmaintainable, buggy and unstable.
The people who try to adapt the technologies that are better suited for large scale deployments for their own needs, might find that the complexity grinds their velocity to a halt and makes them waste huge amounts of effort in maintaining this solution.
The people who want to make someone else deal with these complexities and just want the benefits of the tech, will probably have to pay appropriately for this, which sometimes is a good deal, but other times is a bad one and also ensures vendor lock.
The people who reevaluate what they're trying to do and how much they actually need to do, might sometimes settle for a lightweight solution which does what's necessary, not much more and leads to vaguely positive outcomes.
This is the real value of Kubernetes. I'm at my fourth kubernetes shop and the deployments, service, ingress, pod specs all look largely the same. From an operations standpoint the actual production system looks the same, the only difference between companies is how the deployments are updated and what triggers the container build and how fast they're growing. Giant development/engineering systems went from very specialized, highly complex systems, to a mundane, even boring part of the company. Infrastructure should be boring, and it should be easy to hire for these roles. That's what Kubernetes does.
Oh, and it scales too, but I'd say 95% of companies running their workloads on k8s don't need the scaling it is capable of.
Folks here are making sound arguments for swarm and alternatives and they are totally valid. But for an experienced engineer who needed to start their homelab back up after a long hiatus, I will never look back once for choosing k8s. Honestly some of the most fun I've had with computers in a long time and great experience for when I move into a k8s related $job.
Only?
I have had a home server for fifteen years and used to have a homemade router. That's significantly more time than I have ever spent on them. I don't even bother maintaining scripts for my home installation. I have reinstalled it once in the past decade and it took me significantly less time than writing said scripts would have. There was enough things I wanted to do differently that any scripts would have been useless by that point anyway.
At that point, it sounds more like a hobby than something saving you time.
That's not my experience at all. Plenty of web apps for self hosting are pretty simple. You just install the packet, read the config files, turn on and off what you need and you are good to go. Since systemd made that easy, most apps now come preconfigured to use a dedicated users and drop the capabilities they don't need. If they don't, you are two lines aways from doing it in their service file.
Then I just throw Caddy in front of everything I want to access from outside my local network and call it a day.
Or perhaps one of the lightweight ones, like K3s ( https://k3s.io/ ) or k0s ( https://k0sproject.io/ )?
Maybe a turnkey one, like RKE ( https://rancher.com/products/rke/ )?
Or did you use something like kubespray ( https://kubespray.io/#/ ) to create a cluster in a semi automated fashion?
Alternatively, perhaps you built your cluster from scratch or maybe used kubeadm ( https://kubernetes.io/docs/setup/production-environment/tool... ) and simply got lucky?
Out of all of those, my experiences have only been positive with K3s (though i haven't tried k0s) and RKE, especially when using the latter with Rancher ( https://rancher.com/ ) or when using the former with Portainer ( https://www.portainer.io/solutions ).
In most of the other cases, i've run into a variety of problems:
- different networking implementations that work inconsistently ( https://kubernetes.io/docs/concepts/cluster-administration/addons/#networking-and-network-policy )
- general problems with networking, communication between nodes, and certain node components refusing to talk to others and thus nodes not allowing anything to be scheduled on them
- large CPU/RAM resource usage for even small clusters, which is unfortunate, given that i'm too poor to afford lots of either
- no overprovisioning support (when i last tried), leading to pods not being scheduled on nodes when there's plenty of resources available, due to unnecessary reservations
- problems with PVCs and storing data in host directories, like how Docker has bind mounts with few to no issues
- also, they recently stopped supporting the Docker shim and now you have to use containerd, which is unfortunate from a debugging perspective (since a lot of people out there actually like Docker CLI)
That said, when i compared the RAM usage of Docker Swarm with that of K3s, the picture was far better! My current approach is to introduce clients and companies to containers in the following fashion, if they don't use the cloud vendors: Docker --> Docker Compose --> Docker Swarm --> (check if there's need for more functionality) --> K3s.
The beautiful think is that because of the OCI standard you can choose where and how to run your app, as well as use tools like Kompose to make migration very easy: https://kompose.io/
It's not like cert-manager does rocket science that is possible only on k8s
> Why not have ingress intrinsically linked to the underlying reaources like k8s does? Your suggestion means that someone needs to come up with a methodology and automation to update ingress with Swarm. Yeah, it's obviously possible but imo, Traefik ingress reaources are way easier to manage with config spread across far fewer places.
Traefik can provide a lightweight ingress for Docker Swarm clusters alongside ensuring most of the certificate related functionality that you might want for web apps: https://traefik.io/
In particular, here's an example of using it with Docker Swarm: https://dockerswarm.rocks/traefik/
It integrates well enough that you can set up listening for particular ports, domain names, set up basicauth, use Let's Encrypt for certificates and do many other things with labels inside of your Docker Compose file!
> Storage.. I have an nfs storage provider in k8s which allows me to dynamically create storage with lifecycle management included... for free... just because I'm using k8s.
Docker also supports volume plugins, some of which were carried out of the base offering not to create needless bloat (much like what K3s does with many other in tree plugins of the heavyweight Kubernetes distros): https://docs.docker.com/engine/extend/plugins_volume/
Here's a pretty good documentation page on how to set them up and use them: https://docs.docker.com/storage/volumes/#share-data-among-ma...
You'll find examples of SSHFS, NFS, CIFS/Samba there, which should cover most base use cases. The latter two are supported by the default "local" driver (which feels like a naming failure on their part), so it's safe to say that Docker and Docker Swarm support both NFS and CIFS/Samba out of the box.
> DR as in something happens to my house or other technical issues which bring down my cluster. I would posit that it's much easier for me to restore a cluster because it's all yaml and imo more descriptive and capable than swarm yaml syntax.
I'm afraid i don't follow here.
Here's how you set up a cluster: https://docs.docker.com/engine/swarm/swarm-tutorial/create-s...
There are also commands for managing nodes within the Swarm: https://docs.docker.com/engine/reference/commandline/node/
There are also commands for managing the stacks that are deployed: https://docs.docker.com/engine/reference/commandline/stack/
That's literally it. Verbose YAML doesn't imply any additional quality. If you want, feel free to look at the Compose specification too see all of the supported options: https://docs.docker.com/compose/compose-file/compose-file-v3...
With Swarm, i've actually found that it's actually very easy to both manage the cluster, resolve any issues (because of very few moving parts), or even just wipe it all and create it anew with all of the deployments in a few commands. Furthermore, there is a very lovely Ansible Swarm module if you'd like: https://docs.ansible.com/ansible/latest/collections/communit...
In short, fewer moving parts equal easier administration and fewer Byzantine failures in this case.
> I think you get my point. Yes, you can do it all with Swarm but with things like k3s and k0s, it's just as easy to start using k8s. And way more help, docs, and tooling to boot.
K3s and other projects like it are a step in the right direction for Kubernetes in smaller deployments! That said, in my experience, the documentation of Docker Swarm is comparatively better to that of many of the Kubernetes distro projects, simply because of how mature and stable Docker and Docker Swarm both are.
I don't doubt that the situation will change in a few years, but until then, Swarm is more than enough.
My problem with swarm is that the overlay networking was super flaky. Dead containers still being registered on the network cause every X requests to fail (since it’s round robin) because it’s routing to an overlay IP that doesn’t exist.
Further, I could never find storage adapters for cloud storage that continued to be maintained. Rexray was the longest standing but hasn’t had a commit since Jan 2019.
I will say that it's still a fair amount of complexity, but it's a workable amount of complexity on a day-to-day basis.
I run bitwarden, syncthing, transmission etc. on a single box at home and this setup works great.
Overall, though: Traefik proxying to Docker Containers has been _very_ smooth in my Homelab: I'm loving it.
- pick up Swarm or Nomad now, and then potentially be blamed for the expensive and costly migration to k8s - pick up k8s now, waste the time of your (voiceless) devs and ops on an overengineered solution, but retain access to the larger pool of technologies and a smooth upgrade path should the need arise
The kind of reasoning you pose is similar to "SQL doesn't scale to infinity, so start with an infinitely scalable eventually consistent document store". This line of thinking is dangerous to most companies, and often the death of startups. Assume YAGNI.
There's some truth to "Nobody ever got fired for choosing IBM", but IBM was never a good choice for anyone except consultants. k8s isn't too far from that.
Anyone can leann k8s basic principles in a day or 2. And managing a cluster is arguably easier than managing a typical bunch of EX2 instances+ALB etc that almost everyone uses.
k8s also has the advantage of ubiquity. If I had to manage a bunch of containers on a cluster with > 10 nodes, I'd pick k8s over swarm/nomad etc.
Your argument is 'k8s isn't a good choice for anyone except Google-scale' and I don't think thats true at all.
As someone who has actively avoided k8s (despite enjoying devops), my observation has been quite the opposite. At my current (small) org we have one devops guy who spends all day working on k8s related work (configuration, troubleshooting, automation, etc.) And we're about to start looking for another devops engineer because he's swamped. We only have a handful of clusters up and running so far.
k8s is the poster child for overcomplicating matters for folks that don't have Google-scale problems.
There are problems that people run into. One is not needing more than one computer -- if everything you need to run fits on one computer, you might as well use systemd as your orchestrator. (Which people also say is overly complicated, but really it's just extremely misunderstood.) The other is not understanding Kubernetes enough. A lot of people just explode random helm charts into their cluster and are mad that it's hard to maintain. Basically, adding another abstraction layer only helps if you can debug the actual thing on the surface -- otherwise, you're just setting a time bomb for yourself. (And, helm charts require you beg the maintainers for every feature you could possibly need, and once enough people have done that you just have raw k8s manifests where everything is called some random word instead of its actual thing. Really bad. Use Kustomize or jsonnet!)
ECS isn't exactly simple either, but it is manageable for small shops where the developer is the devops guy.
Not to mention that you end up scaling to a size where k8s is a sound solution, that "costly" migration is going to be a rounding error on your balance sheet
This. Just because your services are containerized doesn't mean they'll scale well with each order of magnitude. Likely you'll need to re-architect other parts of the system to work at your new scale.
Kubernetes has a serious growth ceiling - way below what Nomad has. So if you actually have a problem of interesting scale, it's almost certainly "pick Nomad now, or get blamed for a costly migration later when you realise that Kubernetes is not for you".
Docker Swarm remains far simpler to set up, run, and troubleshoot than Kubernetes.
I just can't choose it over K8s, though, knowing that it's unsupported.
My fallback, if Swarm is ever removed, is Hashicorp's Nomad. I've only used it in a hobby capacity, but is (at face value, at least) another wonderful piece of software. I've used k8s a bunch in production (including right now at my current employer) and just can't recommend it – it's a technical distraction for most teams.
Borg is great if you have Google problems. K8s is not Borg.
That said, if your tooling or products or software supports kubernetes, which a lot more are these days, then it becomes a lot easier. A lot of companies these days are saying "Here's how to install it, or just grab this helm chart"; Gitlab, for example. We even had one piece of software we were testing which only had two options: cloud-hosted, or self-hosted on k8s. Nothing else was supported.
k8s doesn't seem to be any easier to configure, with the possible exception of microk8s (which has its own issues), but the amount of things you can do with it once it's up and running is compelling.
It is very simple to spin a minimal distro based cluster. It is relatively simple to have it up and running your solution. Sure getting Kubernetes from the source and tailoring all the required components to run cluster with full HA etc' is a complex stuff, but building a private cloud based on vanilla Kubernetes isn't the only or the most common use case.
You can read Docker Swarm's docs end-to-end in < 30 minutes, and have a cluster running in roughly the amount of time it takes to install Docker on three boxes.
Docker's failing was a failure to market Docker Swarm. Most people don't even know it exists, yet every single machine with Docker has it already! The last time I checked, they only had a single dev that was part-time allocated to supporting it.
So it's not so much that Docker failed to jump on the k8s bandwagon, so much as they built a wonderful thing that they never marketed and semi-abandoned, to focus on Docker Hub and other core Docker things they thought they'd be able to more directly monetize.
Fortunately, Docker Swarm remains a wonderful piece of software, even without particularly active investment, because it is a fairly complete offering for most use cases. And it seems to be growing in adoption.