For those who a) use Proxmox to b) manage containers c) some of which are of ostype: unmanaged (e.g. those who run a virtual OpenWRT router, like me...) be warned that the current version of Proxmox will not start those containers due to a bug/an omission in the PVE::LXC::Setup package (/usr/share/perl5/PVE/LXC/Setup.pm) which makes it attempt to call a non-existing method to configure cgroupv2 support on such containers. The following patch will work for now while the problem has not yet been solved in the repo:

   --- /usr/share/perl5/PVE/LXC/Setup.pm.org
   +++ /usr/share/perl5/PVE/LXC/Setup.pm
   @@ -424,6 +424,8 @@
    sub unified_cgroupv2_support {
        my ($self) = @_;
    
   +    $self->{plugin} //='PVE::LXC::Setup::Base'; # unmanaged
   +
        $self->protected_call(sub {
           $self->{plugin}->unified_cgroupv2_support();
        });

I hit upon this snag when updating the server-under-the-stairs which went off without any problems except for the router container not starting.

Thanks for the heads up! I’ll wait for the .1 release to avoid messing up videos for my bossy toddlers.

My absolute wish is to run Proxmox on a laptop and have a VM drive the laptop display.

Currently this is not possible.

There are many guides on passing through the integrated or dedicated GPU, but NOT using the laptop display in a guest VM. The host/Proxmox doesn't need the display. I want the guest to own it. No looking glass no spice no gl window no virt-gpu. The VM should directly own and exclusively operate the gpu and laptop display.

My dream setup is Proxmox being very minorly configured - a stable base that's easy to restore. Within it 5 VMs: firewall, primary-linux, rescue-linux, linux-for-containers, and Windows 10.

I recognize most folks are running Proxmox on a desktop but I have a very nice laptop. I want to isolate my firewall into a guest VM (opnsense). I want to run my primary Linux in another guest VM - with the Intel iGPU passed through and using the laptop display. I want to run Windows - when I need it - for games that intentionally disallow VMs. I want to run a separate Linux (Rancher) for containers/development. Finally I want a rescue-linux so if the primary fails for whatever reason, Proxmox would fail over by starting the rescue VM - just something with enough for me to remote into the host/proxmox and fix things. Windows would be viewable from RDP or looking glass. The container Linux would be ssh-administered. Same as firewall or web admin.

I LOVE the idea of Proxmox making it easy to backup/transfer/experiment with different OS's.

Because I can't get Proxmox to drive the laptop display - something not possible on AMD or Intel afaik - I settled on NixOS as the host with explicitly configured everything. Windows in a VM. Docker available on the host. It's the best I can do :(

Someday I hope hypervisors-against-metal on laptops is commonplace.

Seriously: If someone can get the laptop display to work from a guest VM on any laptop I would be so happy. I hope AMD gets this because I am so done paying for baby steps with Intel.

EDIT: Looks like I'm trying this again tonight: https://github.com/patmagauran/i915ovmfPkg

(purportedly a romfile for the igpu that _will_ drive the laptop display)

I've tried it before about a month ago but it looks like I was using the "2.1" release from last year. A few things have been updated in June/July.

I think this is how Qube OS works, but I haven't tried it personally so can't be sure. Perhaps check it out - it might be what you're after? https://www.qubes-os.org/

People install debian and then setup the proxmox on top of that with an install script to achieve what you want.

I assume the intention is to run only a bare-bones OS + Proxmox directly on the laptop with all the other functionality virtualised or compartmentalised through KVM and LXC, managed by Proxmox. Running the desktop directly on the hardware does not fit this profile.

I'd say this - having nearly everything compartmentalised with access to the laptop display - might be possible by using containers instead of virtual machines. This would not work for Windows but it could probably be made to work for those mentioned Linux installs.

This sounds pretty cool, but for now the closest thing I’ve tried is Qubes :( being able to actually pass the GPU to a VM would be great. Certainly it is technically possible? Maybe not…

> I want to run Windows - when I need it - for games that intentionally disallow VMs

I don't understand this.. You want to run Windows in a VM for games that disallow VMs?

I dunno, this seems like such a complicated setup... what do you gain from all this? Might as well just run Windows + WSL2.

I did contradict myself there.

Running some games in a VM within VMware or Virtualbox is detected as such. Running a game in a qemu VM there are some options available to hide that the hardware is within a VM - like how the CPU identifies itself. I'm aiming for the setup where the GPU is passed through from the host for exclusive control by the VM. I honestly don't need this RTX2060 except for games - the Intel iGPU is excellent for webdev.

I have an Intel iGPU and a GTX 1070 myself, and currently just dual boot to separate SSDs (the larger SSD for Windows, the smaller for Linux).

When you switch to the Windows VM, do you intend to suspend the Linux VM, or you just want to let it run in the bg?

Also, couldn't you just run the VM in Linux and get the same benefits? Some people are passing through their graphics cards and reporting good performance IIRC.

You're not alone my friend, I also have my old but powerful gaming notebook as a Proxmox host, and I'd very much like to be able to bring it to LAN parties and just boot a Windows 10 and play Age of Empires II while maintaining all my VMs intact.

Can you expand on your use case a bit? Why do you need multiple VMs?

I work in security and I test a lot of things in virtual machines.

Oh you're not alone here. I've often wanted to do that too. It would make certain things easier toast up.

It was released back in July...

(And was shipping Debian bullseye before _that_ was fully released, so they must have had a lot of confidence in it!)

The debian installer was holding back bullseye release, proxmox uses its own installer (and kernel, qemu) so minimal risk as debian was frozen.

Mad fan of Proxmox. I reeeeally wish they would build support for Docker containers.

Yes, I can run them in a VM and currently do just that.

There’s actually three ways to run Docker on Proxmox: in a VM, in an LXC container, or directly in Proxmox itself. I wrote about it here: https://danthesalmon.com/running-docker-on-proxmox/

you can run docker inside lxc as well

  lxc.apparmor.profile: unconfined
  lxc.cgroup.devices.allow: a
  lxc.cap.drop:

Aww yeah! For real? I can't wait to try this out.

Note that it doesn't play well with ZFS, but see here for workaround: https://old.reddit.com/r/Proxmox/comments/m9ce8s/got_docker_...

Can anyone comment on update/package stability when using the free version of Proxmox. It's my understanding that only the paid license allows access to stable repos. I find the community tier price a bit too expensive for my current home use case (mostly nas + plex).

Been running the free version for a couple of years now. No problems at all. I update and reboot it maybe once a month, or whenever there's a severe security update (that I don't miss). It's uneventful.

Same. I’ve been running it a little over 2 years for home/family services and it’s been entirely stable for me. I update slightly less often than parent comment, but have had zero issues from updates.

Likewise, have run it for years on a bunch of machines, never had a problem. I think I've got at least one that's been dist-upgraded all the way from a 4.x release to the current 7.0.

Your understanding is correct. See here: https://pve.proxmox.com/wiki/Package_Repositories

We run Proxmox for non-production-critical VMs. Nothing fancy, just Proxmox hosts, iSCSI storage and standard VMs on that (Windows and Linux).

Every now and then we update hosts using the No-Subscription Repository. No issues from that so far.

Maybe if we updated more frequently we would see more issues. I think we only do it once or twice between each new official version release. That's out of laziness/lack of time, not out of fear of issues from less tested packages.

Have been running it as a Debian system (realized I don’t need Proxmox, but already had some stuff set up) for about a year and had no idea. Absolutely no problems.

one day we'll get native Docker support... love the distro otherwise

Have they said anything about this possibly ever coming to the platform?

this seems like such a really good baseline for a home-cloud. there's such good host infrastructure here. i'm such a huge Debian lover but there's so much more required to turn it into a decent host environment, and Proxmox covers so much ground to do just that.

but i still want Kubernetes as my primary interface atop it.

since LXC is the primary container system on Proxmox (and a damned fine one at that), it seems like one might be able to make something like LXE[1] run, as a shim, to provide a Container Runtime Interface & start running kubelet (the thing that runs containers) atop that.

[1] https://github.com/automaticserver/lxe

Just out of interest, why do you want to use Kubernetes in a home cloud environment? Given the limited scale of such an environment I don't see the need for something like Kubernetes here, just have those containers managed by Proxmox and be done with it. That is how I use it, with task-specific containers (auth, mail, (data)base, a few build environments, p2p, media, reverse web proxy, router, etc) directly managed through pct (Proxmox LXC CLI).

so, you don't have to recreate the wheel.

i.e. I can easily bring up a set of pods in kubernetes that have health checks, such that kubernets will kill a container and restart it automatically if the health check fails.

To do that in proxmox with VMs is a lot more work. In reality, at minimum, proxmox should have the ability to provision a kubernetes environment by provisioning a set of VMs and building a kubernetes cluster out of the VMs, even if it didn't provide a web UI to it. (think GKE or the like). It should enable you to scale up/down and use the proxmox storage for volumes. this would be a MVP for most people to be able to use it at a small scale. One would then want to start work on how one can control ingress into the kubernetes cluster in a standard way.

I also don’t get why Kubernetes would be in any way needed in such setup.

A lot of people don't see. I think that trying to turn a blind eye to complexity is a foolish mistake. There's so much you're given, that helps so much with out of the box Kubernetes. Logging & load balancing/services, and gateway/ingress, and daemon-sets and... how convinced are you that you understand these features well enough to confirm that you don't & wont need them?

It also sounds exhausting to me to try to start from the basis that's what's good enough for everyone is too much for me. That, because we are scared of complexity, we're going to go bake our own systems, go back & re-assess less-popular less common less comprehensive options.

And how happy are you knowing that the ops you do will be unintelligible to 90% of engineers? To me, getting to participate & share & work with other cloud native projects, to get to explore pull in the future, as it is happening, is an option I would give up for nothing.

I also highly recommend this other thread I have, on how home-computing has never ever had a basis to stand from before, and how now that the computing community finally has it's first chance ever to grow good at ops together[1].

But I see enormous vast huge waves of radical retro-conservatism. "I don't know if I need Kubernetes" is a critical mindset that I think is worth assessing, but honestly, 99.999% of the time, you should use Kubernetes. It works fantastically well at all levels of scale, you can decide as you go what pieces you're going to want to make use of, there are super easy get-going-get-containers-started resources, you will have far far far more chance to bring in & use potentially helpful emerging technology. Please everyone: don't fucking decide to hike the fuck off to your stupid ass cabin in the woods computing environment cause you're all "I don't think this is needed." You are punking yourself. Asking what you need is a question you can harp on for a long time, but the risk of starting with the good comprehensive, proven, well-integrated, robust, scalable (low and high), known, well used/well liked not hard to start with default option is- I think- a low enough that you should probably just give it a spin.

[1] https://news.ycombinator.com/item?id=28332604

I was referring to your specific use-case. I can go to k8s’ website to check its features, no need to list them here. It seems you are trying to create your own SaaS/PaaS to manage your home infrastructure. However, I still don’t see k8s as proper tool for this, because the scale is not big enough for such. Remember, k8s was designed with the Borg philosophy. But maybe you have a special case, so by all means, go for k8s.

I don’t think anyone is scared of complexity. However, not everyone has easy horizontal scalable web applications. And most of time one just wants environmental organization and isolation, which containers/VMs brings. In home clouds I would see vertical scaling as a critical feature for efficiency in the limited infrastructure that cannot be easily scaled up. And today, k8s vertical auto-scaler isn’t anything but a disguised horizontal auto-scaler.

Proxmox isn’t competing against Kubernetes, because by design k8s doesn’t support containers that run anything other than a single application/process that is supposed to be easily killed.

If anything, LXD is the tool now competing with Proxmox due to its recent support to Qemu virtual machines. Although I don’t see what super advanced use case you have, I agree it is okay to run k8s on top of your Proxmox cluster. And you should check LXE limitations.

Have you checked out https://kubevirt.io

An operator for kvm virtual machines that run on the kubernetes nodes.

Kubernetes instead of the proxmox html5 UI

Proxmox is in all honesty the best Virtualization platform out there.

Personally I much prefer OpenStack. Probably overkill for most though.

I had a good experience with Proxmox myself, though Lxd seems to be something of a new competitor now that it supports both containers and VMs.

I upgraded my three-node homelab cluster to Proxmox 7 a while back and everything worked perfectly for me.

(July 6, 2021)