Proxmox Docker Containers Monster – 13000 containers on a single host

Re:Proxmox Docker Containers Monster – 13000 containers on a... by galbar on Monday April 17, 2023 @10:27AM (#35601223)

>docker swarm init

And just like that you have a cluster to run containers on. I really like the simplicity of Docker Swarm. I've been using it for at least five years and it's just worked.

During the COVID lockdown I got tired of having to open a UI (at the time I was using CapRover[0]) to edit any of the services I run so I decided to make my own PaaS with a nice CLI. Connecting to the docker socket is easy and the API is simple enough. It's been working no problem for the last two years.

The only complaint I have is that I can't see the user's IP for HTTP requests[1] but there is some hope in the form of Proxy Protocol[2]. I have no idea how complex the code for Docker Swarm ingress is, but I may spend a weekend in the near future scouting the code to get an idea. The current possible solution is to put a load balancer in front of the cluster that either sets the X-Forwarded-For header (or any of the equivalent ones) or speaks Proxy Protocol but I will avoid that solution for now.

I recommend Docker Swarm as a solution for anyone starting that doesn't want to spend hours and hours configuring a production environment. Even if it is just one node, you get services, replication, healthchecks, restart policies, secrets... And it all starts with that simple command, no further config needed.

[0]: https://caprover.com/ [1]: https://github.com/moby/moby/issues/25526 [2]: https://github.com/moby/moby/issues/39465

[ Parent | on HN ]

Re:Proxmox Docker Containers Monster – 13000 containers on a... by MuffinFlavored on Monday April 17, 2023 @12:05PM (#35602749)

Why Swarm over Compose?

Why Swarm over k3s?

[ Parent | on HN ]

Re:Proxmox Docker Containers Monster – 13000 containers on a... by galbar on Monday April 17, 2023 @12:16PM (#35602894)

compose has no proper concept of service and its replicas. Deploying an update to a service in Compose can be problematic. Swarm also enables multi-node deployments, which you can't do with Compose.

k3s, as simple as it is compared to full k8s, still carries some of the complexity of k8s. Swarm just feels simpler and easier to manage.

[ Parent | on HN ]

Re:Proxmox Docker Containers Monster – 13000 containers on a... by honeybadger223 on Monday April 17, 2023 @12:48PM (#35603330)

Eh....kind of. There's several problems with Swarm in production, even at a small scale. I've ran into issues with nodes refusing workloads, inability to attach to the cluster, Docker failing to start, etc. Also since it's so sparingly used, I have to dig into the depths of Google to try to remediate the issue if I can't figure it out on my own.

Making Swarm production ready is also a time sink. Most people end up with a bunch of bash scripts mashing together YAML for secrets, deployments, etc.

Personally I think it's a waste of developer time. Most people don't need the features of Swarm or K8S. Just use a VM at a popular IaaS provider and stick a LB in front of it. Then you don't have to ask questions about IP rate limiting, or other gotchas. If you outgrow a couple of VMs, we can talk about throwing ASGs or containers at the problem. Honestly though most people I talk to don't need containerized solutions.

[ Parent | on HN ]

Re:Proxmox Docker Containers Monster – 13000 containers on a... by tracker1 on Monday April 17, 2023 @01:44PM (#35604016)

You can absolutely start simpler, with even a single docker host and something like dokku. Then it's a matter of a git push for deploys.

[ Parent | on HN ]

Re:Proxmox Docker Containers Monster – 13000 containers on a... by remram on Monday April 17, 2023 @12:50PM (#35603347)

As opposed to "minikube start"? Or "k3s server"?

[ Parent | on HN ]

Re:Proxmox Docker Containers Monster – 13000 containers on a... by tracker1 on Monday April 17, 2023 @01:43PM (#35603999)

I've been using a relatively small vm with caddy to act as a reverse proxy, this can pass the client IP (X-Forwarded-For) down into the relevant services. I'll usually wrap a logger in the context, so request id and client ip are used, and a request id is passed downstream to peer services per request as well.

[ Parent | on HN ]

Re:Proxmox Docker Containers Monster – 13000 containers on a... by GordonS on Monday April 17, 2023 @03:30PM (#35605368)

I've done this before too. Another approach is to front the site with Cloudflare, which passes the original IP in an HTTP header.

[ Parent | on HN ]

Re:Proxmox Docker Containers Monster – 13000 containers on a... by rzzzt on Monday April 17, 2023 @02:13PM (#35604379)

I liked the Rancher 1.x runtime (Cattle) for similar reasons. It was a little bit more complicated than Swarm, but also offered an IPv4 overlay network that could be accessed by containers across hosts (amongst other things): https://rancher.com/docs/rancher/v1.6/en/rancher-services/

For version 2 they've replaced it entirely with K8s however.

[ Parent | on HN ]