Since k8s is very effective at running a bunch of containers across a few machines, it would appear to be exactly the correct thing to reach for. At this point, running a small k8s operation, with k3s or similar, has become so easy that I can't find a rational reason to look elsewhere for container "orchestration".
It was as much personal "taste" than anything, and I would describe the choice as similar to preferring JSON over XML.
For whatever reason, kubernetes just irritates me. I find it unpleasant to use. And I don't think I'm unique in that regard.
I feel the same. I feel like it's a me problem. I was able to build and run massive systems at scale and never used kubernetes. Then, all of a sudden, around 2020, any time I wanted to build or run or do anything at scale, everywhere said I should just use kubernetes. And then when I wanted to do anything with docker in production, not even at scale, everywhere said I should just use kubernetes.
Then there was a brief period around 2021 where everyone - even kubernetes fans - realised it was being used everywhere, even when it didn't need to be. "You don't need k8s" became a meme.
And now, here we are, again, lots of people saying "just use k8s for everything".
I've learned it enough to know how to use it and what I can do with it. I still prefer to use literally anything else apart from k8s when building, and the only time I've ever felt k8s has been really needed to solve a problem is when the business has said "we're using k8s, deal with it".
It's like the Javascript or WordPress of the infrastructure engineering world - it became the lazy answer, IMO. Or the me problem angle: I'm just an aged engineer moaning at having to learn new solutions to old problems.
I also re-investigated containerization - weighing Docker Swarm vs K3s - and settled on Docker Swarm.
I’ve hated it ever since. Swarm is a PITA to use and has all kinds of failure modes that are different than regular old Docker Compose.
I’ve considered migrating again - either to Kubernetes, or just back to plain Docker - but haven’t done it. Maybe I should look at Uncloud?
Especially in-house on bare metal.
You would not be able to operate hundreds or thousand of any nodes without operation complexlity and k8s helps you here a lot.
Was what i was responding to. It's not the app management that becomes a pain, it's the cluster management, lifecycle, platform API deprecations, etc.
With k8s you write a bunch of manifests that are 70% repetitive boilerplate. But actually, there is something you need that cannot be achieved with pure manifest, so you reach for Kustomize. But Kustomize actually doesn't do what you want, so you need to convert the entire thing to Helm.
You also still need to spin up your k8s cluster, which itself consists of half a dozen pods just so you have something where you can run your service. Oh, you wanted your service to be accessible from outside the cluster? Well, you need to install an ingress controller in your cluster. Oh BTW, the nginx ingress controller is now deprecated, so you have to choose from a handful of alternatives, all of which have certain advantages and disadvantages, and none of which are ideal for all situations. Have fun choosing.
And if you want to use more than one machine then you run `docker swarm init`, and you can keep using the Compose file you already have, almost unchanged.
It's not a K8s replacement, but I'm guessing for some people it would be enough and less effort than a full migration to Kubernetes (e.g. hobby projects).
If you have a service with a simple compose file, you can have a simple k8s manifest to do the same thing. Plenty of tools convert right between the two (incl kompose, which k8s literally hands you: https://kubernetes.io/docs/tasks/configure-pod-container/tra...)
Frankly, you're messing up by including kustomize or helm at all in 80% of cases. Just write the (agreed on tedious boilerplate - the manifest format is not my cup of tea) yaml and be done with the problem.
And no - you don't need an ingress. Just spin up a nodeport service, and you have the literal identical experience to exposing ports with compose - it's just a port on the machines running the cluster (any of them - magic!).
You don't need to touch an ingress until you actually want external traffic using a specific hostname (and optionally tls), which is... the same as compose. And frankly - at that point you probably SHOULD be thinking about the actual tooling you're using to expose that, in the same way you would if you ran it manually in compose. And sure - arguably you could move to gateways now, but in no way is the ingress api deprecated. They very clearly state...
> "The Ingress API is generally available, and is subject to the stability guarantees for generally available APIs. The Kubernetes project has no plans to remove Ingress from Kubernetes."
https://kubernetes.io/docs/concepts/services-networking/ingr...
---
Plenty of valid complaints for K8s (yaml config boilerplate being a solid pick) but most of the rest of your comment is basically just FUD. The complexity scale for K8s CAN get a lot higher than docker. Some organizations convince themselves it should and make it very complex (debatably for sane reasons). For personal needs... Just run k3s (or minikube, or microk8s, or k3ds, or etc...) and write some yaml. It's at exactly the same complexity as docker compose, with a slightly more verbose syntax.
Honestly, it's not even as complex as configuring VMs in vsphere or citrix.
https://kubernetes.io/docs/concepts/services-networking/serv...
Might need to redefine the port range from 30000-32767. Actually, if you want to avoid the ingress abstraction and maybe want to run a regular web server container of your choice to act as it (maybe you just prefer a config file, maybe that's what your legacy software is built around, maybe you need/prefer Apache2, go figure), you'd probably want to be able to run it on 80 and 443. Or 3000 or 8080 for some other software, out of convenience and simplicity.
Depending on what kind of K8s distro you use, thankfully not insanely hard to change though: https://docs.k3s.io/cli/server#networking But again, that's kind of going against the grain.
As for grabbing 443 or 80, most distros support specifying the port in the service spec directly, and I don't think it needs to be in the range of the reserved nodeports (I've done this on k3s, worked fine last I checked, which is admittedly a few years ago now).
As you grow to more than a small number of exposed services, I think an ingress generally does make sense, just because you want to be able to give things persistent names. But you can run a LONG way on just nodeports.
And even after going with an ingress - the tooling here is pretty straight forward. MetalLB (load balancer) and nginx (ingress, reverse proxy) don't take a ton of time or configuration.
As someone who was around when something like a LAMP stack wasn't "legacy", I think it's genuinely less complicated to setup than those old configurations. Especially because once you get it right in the yaml once, recreating it is very, very easy.
That’s not bad, but I want to spend more time trying new things or enjoying the results of my efforts than maintaining the underlying substrates. For that purpose, K8s is consistently too complicated for my own ends - and Uncloud looks to do exactly what I want.
The network is complicated by the overlay network, so "normal" troubleshooting tools aren't super helpful. Storage is complicated by k8s wanting to fling pods around so you need networked storage (or to pin the pods, which removes almost all of k8s' value). Databases are annoying on k8s without networked storage, so you usually run them outside the cluster and now you have to manage bare metal and k8s resources.
The manifests are largely fine, outside of some of the more abnormal resources like setting up the nginx ingress with certs.
The clear target of this project is a k8s-like experience for people who are already familiar with Docker and docker compose but don't want to spend the energy to learn a whole new thing for low stakes deployments.
A normal person wouldn't think 'hey lets use k8s for the low stakes deployment over here'.
I'm afraid I have to disappoint you
I have struggled to get things like this stood up and hit many footguns along the way
K3s is just a repackaged, simplified k8s distro. You get the same behavior and the same tools as you have any time you operate an on-premises k8s cluster, and these, in my experience, are somewhere between good and excellent. So I can't imagine what you have in mind here.
"It's still essentially adding few hundred thousand lines of code into your infrastructure"
Sure. And they're all there for a reason: it's what one needs to orchestrate containers via an API, as revealed by a vast horde of users and years of refinement.
...the fact it's still k8s which is a mountain of complexity compared to near anything else out there ?
It seems that way but in reality "resource" is a generic concept in k8s. K8s is a management/collaboration platform for "resources" and everything is a resource. You can define your own resource types too. And who knows, maybe in the future these won't be containers or even linux processes? Well it would still work given this model.
But now, what if you really just want to run a bunch of containers across a few machines?
My point is, it's overcomplicated and abstracts too heavily. Too smart even... I don't want my co workers to define our own resource types, we're not at a google scale company.
current_context: default
contexts:
default:
connections:
- ssh: [email protected]
ssh_key_file: ~/.ssh/uncloud
- ssh: [email protected]
ssh_key_file: ~/.ssh/uncloud
- ssh: [email protected]
ssh_key_file: ~/.ssh/uncloud
- ssh: [email protected]
ssh_key_file: ~/.ssh/uncloud
And you really just need one entry for typical use. The subsequent entries are only used if the previous node(s) are down.You can either specify one of the machine SSH target in the config.yaml or pass it directly to the 'uc' CLI command, e.g.
uc --connect user@host deploy
In addition to deployments, uncloud handles clustering - connects machines and containers together. Service containers can discover other services via internal DNS and communicate directly over the secure overlay network without opening any ports on the hosts.
As far as I know kamal doesn’t provide an easy way for services to communicate across machines.
Services can also be scaled to multiple replicas across machines.
Are you working on this full time and if so, how are you funding it? Are you looking to monetize this somehow?
I’m working full time on this, yes. Funding from my savings at the moment and don’t have plans for any external funding or VC.
For monetisation, considering building a self-hosted and managed (SaaS) webUI for managing remote clusters and apps on them with value-added PaaS-like features.
Basically just configure it with `{service-name}.internal` to find other instances of the service.
Like rescheduling automatically a container on another server if a server is down? Deploying on the less filled server first if you have set limits in your containers?
There is no automatic rescheduling in uncloud by design. At least for now. We will see how far we can get without it.
If you want your service to tolerate a host going down, you should deploy multiple replicas for that service on multiple machines in advance. 'uc scale' command can be used to run more replicas for an already deployed service.
Longer term, I'm thinking we can have a concept of primary/standby replicas for services that can only have one running replica, e.g. databases. Something similar to how Fly.io does this: https://fly.io/docs/apps/app-availability/#standby-machines-...
Regarding deploying on the less filled machine first is doable but not supported right now. By default, it picks the first machine randomly and tries to distributes replicas evenly among all available machines. You can also manually specify what target machine(s) each service should run on in your Compose file.
I want to avoid recreating the complexity with placement constraints, (anti-)affinity, etc. that makes K8s hard to reason about. There is a huge class of apps that need more or less static infra, manual placement, and a certain level of redundancy. That's what I'm targeting with Uncloud.
A proper load balancer or Cloudflare DNS proxy would handle this.
Note that a DNS A record with multiple IPs doesn't provide failover, only round robin. But you can use the Cloudflare DNS proxy feature as a poor man's LB. Just add 2+ proxied A records (orange cloud) pointing to different machines. If one goes down with a 52x error, Cloudflare automatically fails over to the healthy one.
Could be something interesting to integrate though.
What specifically do you mean by ipv6 support?
This question does not make sense. This is equivalent to asking "What specifically do you mean by ipv4 support"
These days both protocols must be supported, and if there is a blocker it should be clearly mentioned.
What do you think about the general approach in Uncloud? It almost feels like a cousin of Swarm. Would love to get your take on it.
1. External requests, e.g. from the internet via the reverse proxy (Caddy) running in the cluster.
The rollout works on the container, not the server level. Each container registers itself in Caddy so it knows which containers to forward and distribute requests to.
When doing a rollout, a new version of container is started first, registers in caddy, then the old one is removed. This is repeated for each service container. This way, at any time there are running containers that serve requests.
It doesn’t say any server that requests shouldn’t go there. It just updates upstreams in the caddy config to send requests to the containers that are up and healthy.
2. Service to service requests within the cluster. In this case, a service DNS name is resolved to a list of IP addresses (running containers). And the client decides which one to send a request to or whether to distribute requests among them.
When the service is updated, the client needs to resolve the name again to get the up-to-date list of IPs. Many http clients handle this automatically so using http://service-name as an endpoint typically just works. But zero downtime should still be handled by the client in this case.
> I’m building Uncloud after years of managing Kubernetes
did you manage Kubernetes, or did you make the fateful mistake of managing microk8s?
When you run 'uc deploy' command:
- it reads the spec from your compose.yaml
- inspects the current state of the services in the cluster
- computes the diff and deployment plan to reconcile it
- executes the plan after the confirmation
Please see the docs and demo: https://uncloud.run/docs/guides/deployments/deploy-app
The main difference with Docker Swarm is that the reconciliation process is run on your local/CI machine as part of the 'uc deploy' CLI command execution, not on the control plane nodes in the cluster.
And it's not running in the loop automatically. If the command fails, you get an instant feedback with the errors you can address or rerun the command again.
It should be pretty straightforward to wrap the CLI logic in a Terraform or Pulumi provider. The design principals are very similar and it's written in Go.
I get that putting the declarative spec in the control plane and having the service autoreconcile continuously is another layer but this is great as a start.
In fact could you not just cron the cli deployment command on the nodes and get an effective poor man's declarative layer to guard against node failures if your ok with a 1 min or 1 sec recovery objective?
In the project discord, a user recently experimented with a custom setup that sounds very similar to what you describe.
In fact, a big part of uncloud’s appeal to me is that it also provides powerful building blocks for more complex, custom systems like this, not just the streamlined workflow for simpler, standard cases.
Yeah, you absolutely could and someone on our Discord has written a 30-line bash script that essentially runs a reconciliation loop with the CLI.
Thanks for the point on underselling by calling it imperative rather than declarative!
While I would love to test this tool, this is not something I would run on any machine :/
I wanted to try it out but was put off by this[0]. It’s just straight up curl | bash as root from raw.githubusercontent.com.
If this is the install process for a server (and not just for the CLI) I don’t want to think about security in general for the product.
Sorry, I really wanted to like this, but pass.
[0] https://github.com/psviderski/uncloud/blob/ebd4622592bcecedb...
The alternative is, deploying the script and with it have the uncloud files it needs.
I'm not arguing that a repository is nice because versioning, signing, version yanking, etc, and I do agree that the process should be more transparent and verifiable for people who care about it.
You need to prepare the machine some other way first then, but it's just installing docker and the uncloud service.
I use the `--no-install` option with my own cluster, as I have my own pre-provisioning process that includes some additional setup beyond the docker/uncloud elements.
If you do anything professional, you better choose proven software like kubernetes or managed kubernetes or whatever else all the hyperscalers provide.
And the complexity you are solving now or have to solve, k8s solved. IaC for example, Cloud Provider Support for provisioning a LB out of the box, cert-manager, all the helm charts for observability, logging, a ecosystem to fall back to (operators), ArgoCD <3, storage provisioning, proper high availability, kind for e2e testing on cicd, etc.
I'm also aways lost why people think k8s is so hard to operate. Just take a managed k8s. There are so many options out there and they are all compatible with the whole k8s ecosystem.
Look if you don't get kubernetes, its use casees, advantages etc. fine absolutly fine but your solution is not an alternative to k8s. Its another container orchestrator like nomad and k8s and co. with it own advantages and disadvantages.
Some people would rather build their own solutions to do these things with fine-grain control and the ability to handle workloads more complex that a shopping cart website.
The protocols are bad, as is the tech supporting them.
I need to run on-prem, so managed k8s is not an option. Experts tells me I should have 2 FTE to run k8s, which I don't have. k8s has so many components, how should I debug that in case of issues without k8s experience? k8s APIs change continuously, how should I manage that without k8s experience?
It's not a k8s replacement. But I do see a sweet spot for such a solution. We still run Docker Swarm on 5 servers, no hyperscalers, no API changes expected ;-)
But this goes with assumption that one already know docker compose spec. For exact same reason I'm in love for `podman kube play` to just use k8s manifests to quickly test run on local machine - and not bother with some "legacy" compose.
(I never liked Docker Inc. so I never learned THEIR tooling, it's not needed to build/run containers)
The private container IPs will get NATed to the underlying EC2 IPs so requests to RDS will appear as coming from those instances. The appropriate Security Group(s) need to be configured as well. The limitation is that you can't segregate access at the service level, only at the EC2 instance level.
But I wonder what this solves?
Because I stopped abusing k8s and started using more container hosts with quadlets instead, using Ansible or Terraform depending on what the situation calls for.
It works just fine imho. The CI/CD pipeline triggers a podman auto-update command, and just like that all containers are running the latest version.
So what does uncloud add to this setup?
Your setup sounds like single-node or nodes that don't need to discover each other. If you ever need multi-node with service-to-service communication, that's where stitching together Ansible + Terraform + quadlets + some networking layer starts to get tedious. Uncloud tries to make that part simple out of the box.
You also get the reverse proxy (Caddy) that automatically reconfigures depending on what containers are running on machines. You just deploy containers and it auto-discovers them. If a container crashes, the configuration is auto-updated to remove the faulty container from the list of upstreams.
Plus a single CLI you run locally or on CI to manage everything, distribute images, stream logs. A lot of convenience that I'm putting together to make the user experience more enjoyable.
But if you don't need that, keep doing what works.
Technically I could allow my web proxy to discover my services today already, but I refuse to have Traefik (in my case) running as the same user as my services. I prefer to only let them talk over TCP/IP and configure them dynamically with Ansible instead.
It always amazed me that people used that feature in Traefik or Caddy, because it essentially requires your web proxy to have container access to all your other services. It seems a bit intimate to me, but maybe I'm old school.
I share the same concern as top comments on security but going to check out out in more detail.
I wonder if you integrated some decentralized identity layer with DIDs, if this could be turned into some distributed compute platform?
Also, what is your thinking on high availability and fail failovers?
Uncloud[0] is a container orchestrator without a control plane. Think multi-machine Docker Compose with automatic WireGuard mesh, service discovery, and HTTPS via Caddy. Each machine just keeps a p2p-synced copy of cluster state (using Fly.io's Corrosion), so there's no quorum to maintain.
I’m building Uncloud after years of managing Kubernetes in small envs and at a unicorn. I keep seeing teams reach for K8s when they really just need to run a bunch of containers across a few machines with decent networking, rollouts, and HTTPS. The operational overhead of k8s is brutal for what they actually need.
A few things that make it unique:
- uses the familiar Docker Compose spec, no new DSL to learn
- builds and pushes your Docker images directly to your machines without an external registry (via my other project unregistry [1])
- imperative CLI (like Docker) rather than declarative reconciliation. Easier mental model and debugging
- works across cloud VMs, bare metal, even a Raspberry Pi at home behind NAT (all connected together)
- minimal resource footprint (<150MB ram)
[0]: https://github.com/psviderski/uncloud
[1]: https://github.com/psviderski/unregistry