Configuration switching and rollback mechanisms aren’t exactly reliable with trickier setups, as it doesn’t account for any ephemeral state (like what’s actually in the routing tables), and that stuff cannot be always preemptively declared upfront. I’m afraid that despite a lot of efforts, the only truly reliable method to ensure system will come back is still to deploy-and-reboot.
WRT ephemeral state -- NixOS allows to minimize this. Coupled with impermanence, all non-declared and non-externally-retrievable state is wiped away upon reboot. And if it's not declative and not retrievable, I just don't use it. Homelab allows for a lot of choice in that regard
The only thing I have not solved is password-protected sudo on the target host. I deploy using a dedicated user, which has passwordless sudo set up to work. Seems like a necessary evil.
IMO there is no point in doing that over just using root, maybe unless you have multiple administrators and do it for audit purposes.
Anyway, what you can do is have a dedicated deployment key that is only allowed to execute a subset of commands (via the command= option in authorized_keys). I've used it to only allow starting the nixos-upgrade.service (and some other not necessarily required things), which then pulls updates from a predefined location.
> nix run nixpkgs#nixos-rebuild -- switch --flake .#my-flake-target --target-host nixos@$192.168.x.x --sudo --ask-sudo-password --no-reexec
Alternatively, you can set up a secrets service (like a password manager/vault) and source from that. Difference is where the secrets live (encrypted store or networked service, with all the consequences of every approach), commonality is that they’re fetched at runtime, before your programs start.
I’m currently using deploy-rs, but if I’d redo my stuff (and the only reason I don’t is that I’m pretty much overwhelmed by life) I’d probably go with plain vanilla nixos-rebuild --target-host and skip any additional layers (that introduce extra complexity and fragility).
Just a crude and somewhat haphazard summary but hope it helps.