So like, you are using Proxmox as your main OS/desktop? And then doing application (eg firefox for browsing, etc) things in in proxmox LXC containers?

If so, that is pretty interesting...

Although, using LXD with Ubuntu is totally painless and easy as well..

Actually, it went sort of like this:

I used qubes as a desktop os, but it was a little clunky. Like the copy/paste or moving stuff between the vms.

But the main problem was I couldn't ssh INTO the qubes machine, and I couldn't really run server kinds of things (though maybe it was possible to dig into the networking and the firewall vm).

When I tried proxmox, it was more set up for server kinds of things. For example, all the VMs showed up as machines on my local network. I started to make it a desktop, but then backed off. I decided to leave the debian main os vanilla. Instead I made it a server. And I got away from VMs and use containers.

I do have a macos vm that I remote into though. It's a copy of my laptop which I haven't used as much the last year.

I tried running proxmox on my server for a while but I found it was more complex to do everything and I wasn't gaining any benefits from it since it looked very enterprise and multi machine focused. These days I use fedora coreos with podman/docker. I'm trying out virtual machines using libvirt and then virt manager on my desktop for control but I have not been able to find a way to do bridged networking.

Bridged networking in libvirt can be a bit of a pain but I've found that routed works pretty well. Also - if I'm not mistaken - libvirt's netfilter firewalling isn't available on bridged networks.

Is it possible to ssh in to a vm in a non bridged network? I thought you had to give the vm its own local ip on your network.

Sorry, I didn't see that comment earlier!

Routed turns your virtualization host into a router for a separate network on which your VMs live (afaik). So in order to reach that network you have to define static routes that ultimately lead to the virtual interface set up by libvirt.

When you start your routed network (with net-start), libvirt should set up a static route on the host machine automatically. You can see this route with "ip route". To make it available to all hosts on your network, you can configure a static route on your (physical) network's router that covers the IP address range of the virtual routed network and has the (physical) IP address of your virtualization host set as the gateway.

For example: Assume your virtualization host has 10.0.0.10 as its address in your "real" network and your virtual routed network covers 10.0.1.0/24. You would have to set up a static route on your "real" router that says "route 10.0.1.0/24 to 10.0.0.10".

Libvirt automatically sets up a route in the local routing table of your virtualization host that says "route 10.0.1.0/24 to virbr0" (or whatever your network's virtual interface is called).

Then, if everything worked, you should be able to connect to your VMs in 10.0.1.0/24 directly.

I have been considering grabbing a Librem 14 and switching to Qubes as a daily. Could you elaborate on what you found cumbersome?

Qubes is very "opinionated" about how things work and provides a functional but very tightly-knit product that cannot be easily modified to suit your own needs. You have to accept their choices like a good nontechnical user does - you can't modify/replace components or use a modified security model easily.

For example, with version 3.2 which I used for a while, one has to use the default disk setup (no ZFS or other cool storage tech) with slow 2layer filesystems, no GPU acceleration for applications in VMs, mandatory encrypted backup scheme. Regarding security, Qubes makes some strange choices such giving the regular VM unix user root privileges accessible via simple passwordless sudo.

Qubes runs only on a subset of available hardware (motherboard has to be good enough) so watch out for that. Also interaction with VM manager and graphically intensive applications in VMs was sluggish and internet/firewall/audio would randomly stop working when restarting VMs and require system reboot to fix. Some of these may be better now with newest version 4.x, but I am doubtful.

> Qubes is very "opinionated" about how things work and provides a functional but very tightly-knit product that cannot be easily modified to suit your own needs. You have to accept their choices like a good nontechnical user does - you can't modify/replace components or use a modified security model easily.

It's worth mentioning that Qubes has good reason for this.

A big part of the purpose for their existence is to help protect journalists and others who would greatly benefit from enhanced security, but don't know how to get there themselves.

Qubes provides a very specific product. Whether it helps to "protect journalists and others who would greatly benefit from enhanced security" depends on user level of knowledge and the ways he uses the computer. If the user does not know much about security, Qubes may help to isolate different tasks in VMs, but that is not a panacea. He may still use it wrong in single VM, for example leave the passwordless sudo on.

Serious security-seeking user needs to educate himself about how the computer and internet works, about operational security, pervasive tracking, typical attacks etc, not just use a product and call it a day. Only so educated user can decide if the Qubes with its benefits/drawbacks is worth it.

Of course. But there is a difference between expecting someone to learn what it does, and why it does it, and expecting them to learn how to implement/modify it themselves.

People have work to do, work that has computers as a tool, not a goal. Lowering the bar from "Learn the exploits" to "Be careful and stay in the lines" is a Good Thing, imho.

Qubes provides an effective framework to say "This is how to use it, this is what you should be aware of, don't do this".

> graphically intensive applications in VMs was sluggish

Just as a datapoint youtube or vlc seem to work well enough. The sluggishness is definitely noticeable though. As a developer it encourages me to optimize the display performance of my web applications. If it performs ok on Qubes it's liquid smooth on my 3+ year old android phone. Take that as you will...

> internet/firewall/audio would randomly stop working when restarting VMs

I started with 4.X and I haven't noticed this. I never actually used 3.X so I can't say if it's something that was fixed or never present on my hw.

> provides a functional but very tightly-knit product that cannot be easily modified to suit your own needs.

This is a fair assessment. But I've come to realize I need the isolation Qubes provides more than I thought I would. No advanced threats or anything. Just the ability to put email, chat, password manager, and work in separate reproducible environments is really cool. Clicking a url on a chat won't open in the browser I'm signed into google or in my work browser. If you have the need for gpg or tor they offer solutions with a high degree of isolation, though I haven't looked into them much.

> Just as a datapoint youtube or vlc seem to work well enough.

As another datapoint, when I installed Qubes about six months ago on a i6600k with 16GB RAM, in the default VM and browser, YouTube was basically unwatchable.

> Regarding security, Qubes makes some strange choices such giving the regular VM unix user root privileges accessible via simple passwordless sudo.

See here for details: https://www.qubes-os.org/doc/vm-sudo/

You can replace passwordless sudo with a secure user prompt: https://www.qubes-os.org/doc/vm-sudo/#replacing-passwordless...

Notice how that argument starts with the conclusion "In Qubes VMs there is no point in isolating the root account" and proceeds to rationalize it, very poorly, by using some cherry picked examples where that may be true.

OK, an attacker capable to attack the hypervisor when running as root may be often "game over" if it runs as regular user. But that kind of attack is so rare, why is it the base of the argument at all?

Meanwhile your browser/other big SV application/script from Github can become root whenever it wants, or by accident. It can send contents of the whole VM over the Internet to anybody. Or an honest bug in it can destroy your whole VM. This is why the unix privilege separation or even more capable MAC systems exist. Seriously, running untrusted applications with root access is a braindead idea.

In Qubes, you typically don't care if your untrusted application compromises the (untrusted) virtual machine. This is the whole point of the virtualization and separating your work into security domains. See also: https://xkcd.com/1200/.

Anyway, Qubes team is not against such kinds of isolation: https://qubes-os.discourse.group/t/isolation-within-the-same....

> In Qubes, you typically don't care if your untrusted application compromises the (untrusted) virtual machine.

For that to be true, one would have to run each application in a dedicated VM. A great idea, but Qubes does not recommend that for efficiency reasons. It does pose performance and usability challenges which is why most people do not use it that way. So yes, you do care about applications in single VM not snooping/manipulating each other. Luckily there are standard unix and MAC mechanisms to isolate them.

You don't run each application in a dedicated VM, but you have security domains, each of which have equally (un)trusted applications.

Also, there are disposable VMs for truly untrusted things.

> I have been considering grabbing a Librem 14 and switching to Qubes as a daily

FWIW, I did the same thing and was unhappy with the result. The Librem hardware wasn't great quality and Qubes really requires a massive desktop to run well. And if you are paranoid enough to need Qubes, then you want ECC memory to defend against side channel attacks.

Not sure what you are talking about. Any details? Qubes runs smoothly on my Librem 15 with 32 GB RAM.

proxmox security is pretty thin last time i checked. everything in dom0 runs as root...