I've used this[1] Terraform provider together with the Talos[2] distribution for deploying a Kubernetes cluster. I agree that the APIs available with Proxmox are not fully featured, but it more than suits my needs.

I'm running a four node cluster on salvaged SFF machines backing up lvm snapshots to home brewed TruNAS storage and it all makes me happy.

----

[1] https://github.com/Telmate/terraform-provider-proxmox

[2] https://factory.talos.dev/

You probably know this but it's good to run a cluster with an odd number of nodes. You don't even need another full node, just a quorum node like a RPi.

Yes, of course.. I'm actually in the process of replacing nodes. The original 3x Ryzen5 4-core 32Gb hosts are being replaced by Ryzen9 12-core 96Gb hosts.. its just taking a bit of time. As long as I only ever take one down for updates at a time, its no bother for a home-lab environment.

I've pretty thoroughly drunk the NixOS Kool-aid.

For awhile I ran Docker Swarm with a bunch of SBCs, then k8s, then just a big server running Ubuntu + Cockpit, then Proxmox, until I have finally settled on NixOS.

NixOS has decent container support if necessary, but I've found that its declarative nature means I almost never bother with containers. "Uninstalling" something is generally as simple as "remove it from the config file, rebuild", and it's not hard to do cgroupey stuff if you need to manage memory and the like.

Not to mention that I think NixOS's nginx DSL is wonderful. It's so nice being able to have my proxy configs (along with LetsEncrypt) managed directly (and correctly) by the config environment instead of me writing my own scripts and the like.

(I'm not sure if there are any distributed NixOS things, because I could totally see something neat being built on Flakes)

My homelab has never been simpler and I've never been happier with it.

Any reason you didn't go NixOS in a Proxmox VM? The advantage would not be having to do a full reinstall if anything went wrong and being able to spin up other OS' if needed. The downside would be a few percentage of performance loss.

NixOS takes a snapshot on every rebuild, which happens pretty much every time you install something or change a configuration setting, meaning that if I screw something up, generally all I have to do is reboot and choose the previous generation.

Of course I could install NixOS inside Proxmox, but part of the appeal of NixOS is that everything in the system is managed by the configuration.

It's certainly a different model of deployment. I like it, though it does have its warts.

However there is a (community) TF module...? https://registry.terraform.io/providers/Telmate/proxmox/late... (I have no experience with it as I typically reach for Ansible).

Also, easy-to-install ZFS makes it hard for me to cajol myself into trying something else. And if I want k8s for play time I can always spin up (a/some) VM(s).

There is also this one: https://github.com/bpg/terraform-provider-proxmox

I've been automating deployment with the bpg Terraform module linked by a sibling (of mine) comment to Proxmox for work.

Neither option is particularly complete, and they have some issues; the bpg one does most of the heavy lifting over SSH rather than using the API due to missing features; it also has some annoying quirks with data structure, such as VM IPs are in multi-dimensional arrays, which means you have to write a bunch of logic to drop localhost and secondary IPs (such as those for Docker virtual networks), and then restructure the output, if you want to use the address to setup your DNS for example.

It's doing what I need now, but I would not call them "gold" or "platinum" grade, probably "silver".

I'd suggest seeing if Proxmox is better-supported in some other IaC tool and fallback to Terraform as a last resort.

Have you tried incus?

https://linuxcontainers.org/incus/

I moved my Proxmox single node home-prod setup to Incus over the last couple of weeks.

Incus feels a lot less…legacy? Old school? Something.

Not a lot different when it gets down to it though. It’s easier to work at the CLI with Incus. Backups are a little less straight forward.

Proxmox isn't legacy, far from it.

I see Incus also uses/used LXC, which has been my main gripe with Proxmox; I'm intimately familiar with building Docker/Podman images but have never built an LXC.

Now that Incus has shutdown the image server[0] is there a decent source for LXC images? I've often struggled to find ready-made images for a lot of things I want to deploy on Proxmox, and if I was to move away, I'd probably want something that uses Docker/Podman for when I don't want to deploy a VM.

[0]https://discuss.linuxcontainers.org/t/important-notice-for-l...

For people who might be confused, Incus is what used to be LXD. It used to be a Canonical project but people who didn't like their direction forked it an made much easier to install as well. (It was only available on Snap for a long time) I think the main developer uses opensuse so their rpm packages are pretty good.

As for LXD/Incus itself, I sincerely believe it's good software and I like their CLIs a lot more but for my own purposes i've moved to using proxmox, or lxc directly.

Different technologies are for different approaches and applications.

It's relatively trivial to use the pve command line utility to create or modify vms in proxmox.

Still, the originating reason of this post is due to a large number of useful scripts to help make things more manageable and maintainable, and the founder of it having to step away, and there being gratitude for their help to make things much more manageable.

> Still, the originating reason of this post is due to a large number of useful scripts to help make things more manageable and maintainable

Also makes it very quick to try out an application, arguably less time than even docker.

Absolutely.

Docker is a step or two away from packaging installers for the masses.

I hadn't intended to take away from that. And I've used these scripts myself for spinning up resources - they're definitely a help.

But you can just chuck Kubernetes nodes on Proxmox? I have my nodes running on XCP-ng. The beauty of running a hypervisor is maximum flexibility. I can try out different distros etc, either for k8s nodes or otherwise. I run my router on there (opnsense). I can play with stuff like nix and guix and could even install Windows if for some reason I wanted to.

Proxmox doesn’t preclude you from having k8s. You can create VM(s) in Proxmox and then install k8s on them, then run your app workloads in k8s.

You do have to treat Proxmox VMs like “pets, not cattle” since they are more difficult to automate, but that’s the same story as if you were managing your k8s host on bare metal too. The benefit with Proxmox-hosted VMs though is that you can use Proxmox for whole-VM backups and migrations, so you can have the best of both proxmox and k8s!

Something like Talos gets you pretty close to cattle. You just boot a fresh VM from a generic ISO, then run a pre-defined config against it and it will join the cluster. I haven't looked into it but in theory you could pre-bake that config into the boot ISO so adding a new node would literally be just add a new VM using a template. Of course you'd want to remove the node from the cluster cleanly before just deleting it, though.

I run a couple of Talos clusters on Proxmox at home; I haven't templated them yet as they're pretty static clusters, but I suppose you could use a Proxmox Snippet with the config in and point new Talos nodes at it when they boot.

I've also been using a Terraform module for Proxmox at work to deploy stuff, but there's only two, both community modules and neither is gold/platinum tier, good enough for homelab though I'd say.

I have a single VM on my proxmox server that I spin up all my docker containers in... This is the simplest thing I could think of in terms of config. I haven't had to wire containers together, though, so maybe I've found the sweet spot for my deployment needs.