Great! What guests do you typically run where you see better performance with libvirt/kvm?

Mostly Linux, but also Windows when I had a VFIO passthrough setup. I don't think it's even possible to set it up with Virtualbox to have decent enough performance.

You can also setup VFIO in VirtualBox/KVM. We haven't polished it yet though. You can check the video here to see GPU virtualization in action:

https://www.cyberus-technology.de/products/hypervisor (Don't mind the English, we are not native speakers. :)

I'm really curious about this, is it gpu para-virtualization or actual VFIO requiring built in support from the gpu hardware?

We have used this with recent Intel GPUs that support SR-IOV. This is what you see in the video on the cyberus website. Intel hasn't managed to upstream the drivers for this yet and you have to piece together things, which is very unpleasant. But we are there to help, if someone wants to use this in a professional setting.

We used to have support for Intel GVT-g GPU virtualization as well, which was more of a software solution. This doesn't work with modern Intel GPUs anymore.

I wonder if it would be possible to use DXVK here for windows guests to play those unruly games. That is have a pass through direct x driver in windows that sends all the commands to dxvk which either sends the image back to the vm or renders directly on the screen.

Thank you for the info. I use AMD consumer GPUs, none of them unfortunately support SR-IOV afaik, there are some developments from google around virtio-gpu(DRM native context) I've been following, I was hoping this was something similar.

I thought virt manager was ok but honestly your complaints about it are specific and fair.

Virtual box has graphical configuration for a ton of different options. It also “just works” in many cases and is relatively easy to use.

I am surprised the open source community has not built better gui tools, and no project, closed or open has made configuring pcie passthrough easy.

I have always wanted to be able to run Windows in a virtualized session with my GPU for gaming, and use my onboard APU for the Linux host, but the configuration is daunting, and many of the games I play today don’t work on linux thanks to anticheat or DRM.

I wish there was a port of UTM to linux

Gnome Boxes is an attempt at a similar interface, but yeah, it's not quite as polished.

virt-manager supports more complex libvirt XML configurations, can also manage VMs created by Gnome Boxes, but doesn't yet have IOMMU/PCIE passthrough with OVMF UEFI device selector and vm configuration gui: https://virt-manager.org/

> I am surprised the open source community has not built better gui tools

This is the thing that isn't surprising.

In order to make a better GUI tool, you have to understand how the internals work. Then you don't care to use a GUI tool because you know how to use the command line or edit the configuration files. The people who want better GUI tools are the people who don't know how to make them.

In order to change this, the people who want GUI tools but don't know how to make them have to provide some incentive (typically money) to the people who know how to make them but don't want them, e.g. via donations or some kind of commercial agreement. And if they stand around and wonder why nobody else has fixed their problem for them instead of doing the thing that causes it to be fixed, an object at rest tends to remain at rest.

> no project, closed or open has made configuring pcie passthrough easy

"GPU passthrough with libvirt qemu kvm" https://wiki.gentoo.org/wiki/GPU_passthrough_with_libvirt_qe...

"PCI passthrough via OVMF" https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF :

> The Open Virtual Machine Firmware (OVMF) is a project to enable UEFI support for virtual machines. Starting with Linux 3.9 and recent versions of QEMU, it is now possible to passthrough a graphics card, offering the virtual machine native graphics performance which is useful for graphic-intensive tasks

KVM-GPU-Passthrough: https://github.com/BigAnteater/KVM-GPU-Passthrough

https://clayfreeman.github.io/gpu-passthrough/

I don't think that linking two different Wiki's (for different Linux distros) and two different github posts is "easy" compared to VirtualBox's very "fisher price" Next-Next-Next-Done GUI

Not saying I prefer one or the other, but it's worth bearing in mind where "the bar" is

FWICS from scanning those resources, there are a few shell commands to wrap with a config parser and an output parser for a GUI

E.g. virt-manager is built with glade XML and Python:

virt-manager/virt-manager//ui/createvm.ui: https://github.com/virt-manager/virt-manager/blob/main/ui/cr...

virt-manager/virt-manager//ui/gfxdetails.ui: https://github.com/virt-manager/virt-manager/blob/main/ui/gf...

virt-manager/virt-manager//ui/hoststorage.ui: https://github.com/virt-manager/virt-manager/blob/main/ui/ho...

virtManager/createvm.py: https://github.com/virt-manager/virt-manager/blob/main/virtM...

virtManager/device/addstorage.py: https://github.com/virt-manager/virt-manager/blob/main/virtM...

virtManager/device/gfxdetails.py: https://github.com/virt-manager/virt-manager/blob/main/virtM...

virtManager/addhardware.py:

  DeviceController.TYPE_PCI

    def populate_controller_model_combo(combo, controller_type): 

https://github.com/virt-manager/virt-manager/blob/135cf17072... https://github.com/virt-manager/virt-manager/blob/135cf17072...

"Locating the GPU": https://clayfreeman.github.io/gpu-passthrough/#locating-the-... :

  for d in /sys/kernel/iommu_groups/*/devices/*; do
    n=${d#*/iommu_groups/*}; n=${n%%/*}
    printf 'IOMMU Group %s ' "$n"
    lspci -nns "${d##*/}"
  done;

iommu.sh gist: https://gist.github.com/Roliga/d81418b0a55ca7682227d57af2778...

iommu_groups.sh: https://github.com/drewmullen/pci-passthrough-ryzen/blob/mas... :

  lspci -nns "${d##*/}"

"PCI passthrough via OVMF > 2. Setting up IOMMU > 2.2 Ensuring that the groups are valid": https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF#En... :

> An IOMMU group is the smallest set of physical devices that can be passed to a virtual machine. For instance, in the example above, both the GPU in 06:00.0 and its audio controller in 6:00.1 belong to IOMMU group 13 and can only be passed together. The frontal USB controller, however, has its own group (group 2) which is separate from both the USB expansion controller (group 10) and the rear USB controller (group 4), meaning that any of them could be passed to a virtual machine without affecting the others.

"Exporting your ROM": https://github.com/BigAnteater/KVM-GPU-Passthrough?tab=readm... :

  lspci -vnn
  find /sys/devices -name rom

  # PATH_TO_ROM=

  echo 1 > $PATH_TO_ROM
  mkdir -p /var/lib/libvirt/vbios/
  cat $PATH_TO_ROM > 
  /var/lib/libvirt/vbios/gpu.rom
  echo 0 > $PATH_TO_ROM

"Attaching the GPU" [with `virsh`] https://clayfreeman.github.io/gpu-passthrough/#attaching-the... :

  <hostdev mode='subsystem' type='pci' managed='yes'>
    <rom file='/path/to/gpu-dump.rom'/>
    <source>
      <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </source>
  </hostdev>
  <hostdev mode='subsystem' type='pci' managed='yes'>
    <source>
      <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
    </source>
  </hostdev>

"Adding your GPU and USB devices to the VM" [with `virt-manager`]: https://github.com/BigAnteater/KVM-GPU-Passthrough?tab=readm...

> 1. Add every PCI device which has to do with your graphics card to the VM.

That's still MASSIVELY more complex than "Next -> Next -> Next -> Done"

virt-manager plays fine with hidpi on Wayland. On the opposite side, VirtualBox’s GUI is super buggy in Wayland.

It’s basically the opposite for both. I use virt-manager because the GUI is simpler (and setting up virtualbox is a nightmare anyway).

Regardless, this feature is a step in the right direction. I’m wondering if distributions will pick it up or if it will ever be integrated upstream.

That's a very recent change to virt-manager, so recent it's on nixos-unstable, but not on nixos-23.11. So it might be in Fedora 39 and some rolling-release distros... and nowhere else.

Could you link the change? At least a few days ago virt-manager still seemed to have scaling issues with guest displays, on nixos-unstable. I had viewer scaling on though as a workaround, so maybe I just didn’t notice.

It's not perfect. On older versions, it'd sometimes add a black border the size of your scaling factor. It still reports your window size times your scaling factor as internal resolution with guest additions though.

100% agree about the terrible GUI choices. That actually turned me away completely from libvirt for virtual machines because when I actually needed to work with virtual machines I didn't want to finick around with CLIs and I stuck to VMware until recently. (at the time I ran Windows 10 on my main PC and Windows has a pretty terrible CLI)

Have you tried Gnome Boxes?

I have, it is very limited compared to Virtual Box.

Regarding IPv6, there is support. In fact I run IPv6 only networks including for the hypervisors.

    $ virsh net-dumpxml default6
    <network>
      <name>default6</name>
      <uuid>73590ea2-eb15-4e67-b104-319721bdf302</uuid>
      <forward mode='route'/>
      <bridge name='virbr1' stp='on' delay='0'/>
      <mac address='52:54:00:ff:a7:2d'/>
      <domain name='default6'/>
      <ip family='ipv6' address='2001:db8:ffff::1' prefix='48'>
      </ip>
    </network>

One can also use DHCPv6 if required.

In virt-manager?

Virt-Manager does allow you to edit the xml config file by hand from within the interface. So, yes you can use ipv6; but no, it's not a convenient point and click gui interaction.

I'd be curious to hear the specific reasoning behind it.

That's what I meant, I had to use virsh to set that up because it's not possible to do in the GUI.

But it is!

virt-manager v4.0.0 (Ubuntu 22.04 amd64)

"QEMU/KVM" > "Details" (via double-click or context menu) > "Virtual Networks" tab > "+" (Add) button > "Create Virtual Network"/"Details" tab > Expand "IPv6 configuration" > check "Enable IPv6" > enter "Network" prefix and/or check "Enable DHCPv6" > enter "Start" and "End" addresses

If you use a network bridge there's no reason to have to configure any IP addresses at all.

Yeah this has always been a bit of a pain. Since LXD recently launched a web interface that includes graphical console support (plus VM support some time before that, it's not just a container host anymore, it can do both), I have been using that for remote VMs now. But it still has quite a few rough edges, but nicer than libvirt at least.

virt-manager is the kind of software that frustrates me, has lost me VM (due to beraking SPICE beyond it already having been broken for years). Bug reports go ignored. Yet, when it works, it works great. And KVM is great. But Christ, it's a nightmare sometimes. Features lag behind libvirt, sometimes years. Just not great. The virt-viewer port to Windows never made it past POC+1, IMO, also.

I'm honestly surprised there aren't more alternatives. Especially since SPICE has gstreamer instegration. Gstreamer can natively do webrtc video and data back channel. Someone could hack together a webrtc broweser-based VM viewer and I suspect there's a lot of interest, potentially money making things you could crate.