News for hackers. Stuff that matters.
Re:We Replaced Firecracker with QEMU by arun-mani-j on Monday July 10, 2023 @11:18AM (#36667697)
I think you could help me answer the question that has been in my mind for a month :)

Is there any article that tells the difference and relationship between KVM, QEMU, libvirt, virt-manager, Xen, Proxmox etc. with their typical use cases?

Re:We Replaced Firecracker with QEMU by gabereiser on Monday July 10, 2023 @11:25AM (#36667791)
KVM is kernel-based virtual machine, with libvirt being its API abstraction over all of it. QEMU is a virtual machine host that leverages kvm or software virtualization to spin up machines on the host. virt-manager does the same. Xen is another virtual machine host, like KVM. Proxmox is a virtual machine manager (like QEMU, virt-manager) but is web based. Libvirt will provide abstraction for kvm,qemu,xen

Use cases: proxmox web interface exposed on your local network on a KVM Linux box that uses QEMU to manage VM’s. Proxmox will allow you to do that from the web. QEMU is great for single or small fleet of machines but should be automated for any heavy lifting. Proxmox will do that.

Re:We Replaced Firecracker with QEMU by bonzini on Monday July 10, 2023 @11:58AM (#36668249)
This is almost entirely wrong especially as far as QEMU, Libvirt and virt-manager are concerned.

QEMU is a low level process that represents the virtual machine. It has no equivalent in Xen. Using QEMU directly is not a good idea unless your needs for VM configurations change all the time and you hardly reuse VMs.

Libvirt is at a higher level than QEMU. It manages the QEMU processes and gives them access to system resources (image files, network interfaces, pass-through PCI devices). It also makes it easy to manage the configuration of your virtual machines and the resources they use.

Higher still is virt-manager, which is a GUI interface for libvirt. Proxmox sits at roughly the same level as virt-manager.

Re:We Replaced Firecracker with QEMU by gabereiser on Monday July 10, 2023 @12:14PM (#36668467)
How? KVM and Xen are kernel level. QEMU uses KVM but also has a software virtualization capability. Libvirt is an API abstraction over it all. virt-manager is a gui app to manage libvirt machines. Proxmox as well. Proxmox VE talks to VMHost via libvirt.
Re:We Replaced Firecracker with QEMU by bonzini on Monday July 10, 2023 @12:18PM (#36668530)
Libvirt does not use KVM. Libvirt uses either QEMU (which in turn might or might not use KVM) or Xen or other hypervisors. So it's incorrect to say that Libvirt abstracts over KVM.

And virt-manager indeed manages Libvirt machines so it's not at the level of QEMU as you wrote in the parent comment:

> Proxmox is a virtual machine manager (like QEMU, virt-manager)

Re:We Replaced Firecracker with QEMU by gabereiser on Monday July 10, 2023 @12:38PM (#36668865)
Semantics, libvirt abstracts over KVM via QEMU because QEMU/KVM/HVT is all one driver.
Re:We Replaced Firecracker with QEMU by bonzini on Monday July 10, 2023 @02:38PM (#36671015)
KVM is not enough to create a virtual machine. KVM only virtualizes the processor, not a whole machine.
Re:We Replaced Firecracker with QEMU by gabereiser on Monday July 10, 2023 @04:16PM (#36672325)
>Using KVM, one can run multiple virtual machines running unmodified Linux or Windows images. Each virtual machine has private virtualized hardware: a network card, disk, graphics adapter, etc.

Straight from their site. QEMU is the user space interface, KVM the kernel space driver. It’s enough to run whatever OS. That’s the point.

For libvirt: https://libvirt.org/drivers.html

They support a bunch as well.

Re:We Replaced Firecracker with QEMU by bonzini on Monday July 10, 2023 @04:24PM (#36672441)
Exactly, it's QEMU that abstracts over the processor virtualization APIs of KVM, Microsoft WHPX, Apple Hypervisor.framework etc. Not Libvirt.
Re:We Replaced Firecracker with QEMU by cthalupa on Monday July 10, 2023 @05:20PM (#36673115)
I don't want to necessarily make this an argument to/from authority, but for some context here - you are discussing this with Paolo Bonzini, maintainer of KVM, contributor to QEMU. In the list of people that best understand the difference and demarcation points between KVM and QEMU, he's pretty far up there.
Re:We Replaced Firecracker with QEMU by Izmaki on Monday July 10, 2023 @11:45AM (#36668085)
I don't know if _one_ such article exists, but here is a piece of tech doc from oVirt (yet another tool) that shows how - or that - VDSM is used by oVirt to communicate with QEMU through libvirt: https://www.ovirt.org/develop/architecture/architecture.html...

In really simple terms, so simple that I'm not 100% sure they are correct:

* KVM is a hypervisor, or rather it lets you turn linux into a hypervisor [1], which will let you run VMs on your machine. I've heard KVM is rather hard to work with (steep learning curve). (Xen is also a hypervisor.)

* QEMU is a wrapper-of-a-sorts (a "machine emulator and virtualizer" [2]) which can be used on top of KVM (or Xen). "When used as a virtualizer, QEMU achieves near native performance by executing the guest code directly on the host CPU. QEMU supports virtualization when executing under the Xen hypervisor or using the KVM kernel module in Linux." [2]

* libvirt "is a toolkit to manage virtualization platforms" [3] and is used, e.g., by VDSM to communicate with QEMU.

* virt-manager is "a desktop user interface for managing virtual machines through libvirt" [4]. The screenshots on the project page should give an idea of what its typical use-case is - think VirtualBox and similar solutions.

* Proxmox is the above toolstack (-ish) but as one product.

---

[1] https://www.redhat.com/en/topics/virtualization/what-is-KVM

[2] https://wiki.qemu.org/Main_Page

[3] https://libvirt.org/

[4] https://virt-manager.org/

Re:We Replaced Firecracker with QEMU by Izmaki on Monday July 10, 2023 @11:52AM (#36668171)
Ps: typically if you want to run VMs you are faced with two paths only: 1) you want one or more VMs on your personal computer or 2) you want one or more VMs in an office environment.

On the first path you are likely going to be just fine with VirtualBox, VMWare Workstation or Hyper-V (Windows only) / Parallels (Mac intended). Which one you should pick depends on your desired use of the machines.

On the second path you would go with a solution that deals with the nitty-gritty details, such as Proxmox, oVirt, Hyper-V, ESXi, or any of the other many available options - granted you are not going full cloud-based, which opens up a whole lot of different options too.

You would generally never need to worry about which components are needed where and why. I've had to worry about it once or twice before, because I've had to debug why an oVirt solution was not behaving like I wanted it to behave. Knowing the inner workings helps in that case.

Re:We Replaced Firecracker with QEMU by crabbone on Monday July 10, 2023 @03:00PM (#36671315)
> On the first path you are likely going to be just fine with VirtualBox, VMWare Workstation or Hyper-V (Windows only) / Parallels (Mac intended).

As a Linux user, why would you want to use VirtualBox or VMWare Workstation? They are not so well integrated with the system, and, frankly, VirtualBox is more of a toy VM player... just go for virt-manager. It gives a conceptually similar interface to VirtualBox, but better integration with the rest of the system. Especially, when it comes to stuff like sending different key combinations.

I honestly cannot think of a single benefit to using VirtualBox (and I'm less familiar with VMWare player) compared to virt-manager. My guess is that it's more often used because it's also a common choice on MS Windows, so, you get more hits if you are going to search the Web for questions associated to VMs / you'd get tutorials for how to set up a VM that use VirtualBox. But, if you apply yourself to learning how either one of these works, you'd see no reason to choose it.

Re:We Replaced Firecracker with QEMU by mkesper on Monday July 10, 2023 @05:07PM (#36672971)
The only reason for using it would be that the GUI is a little bit more capable. But feature crippling without the PUEL addons is horrible.
Re:We Replaced Firecracker with QEMU by jacobr1 on Monday July 10, 2023 @11:51AM (#36668161)
This is exactly the type of summary chatGPT is great are providing:

Sure, I can provide a brief comparison and relationship between KVM, QEMU, libvirt, virt-manager, Xen, and Proxmox, as well as their typical use cases:

1. KVM (Kernel-based Virtual Machine)

KVM is a full virtualization solution for Linux on x86 hardware containing virtualization extensions (Intel VT or AMD-V). It consists of a loadable kernel module (kvm.ko) that provides the core virtualization infrastructure and processor-specific modules like kvm-intel.ko or kvm-amd.ko.

Use Case: KVM is typically used for creating and running multiple virtual machines (each with private virtualized hardware) directly on a physical server's hardware. It's used in data centers, in cloud infrastructure, and for enterprise-level virtualization.

2. QEMU (Quick Emulator)

QEMU is a hosted virtual machine monitor; it emulates CPUs through dynamic binary translation and provides a set of device models, enabling it to run a variety of unmodified guest operating systems. It can also be used with KVM to run virtual machines at near-native speed (by executing the guest code directly on the host CPU).

Use Case: QEMU is often used in conjunction with KVM in Linux distributions to achieve efficient virtualization. It can also be used standalone for hardware emulation.

3. libvirt

libvirt is an open-source API, daemon, and management tool for managing platform virtualization. It can be used to manage KVM, Xen, VMware ESXi, QEMU, and other virtualization technologies. These APIs are widely used in the orchestration layer of hypervisors to manage the lifecycle of virtual machines.

Use Case: libvirt is commonly used to manage several different hypervisors, providing a common, generic, and stable layer of abstraction to manage these systems.

4. virt-manager (Virtual Machine Manager)

virt-manager is a desktop user interface for managing virtual machines. It primarily targets KVM VMs but also manages Xen and LXC (Linux Containers). It presents a summary view of running domains, their live performance & resource utilization statistics.

Use Case: virt-manager is used in scenarios where a GUI is preferred over command-line interface for managing virtual machines.

5. Xen

Xen is a type-1 hypervisor, providing services that allow multiple computer operating systems to execute on the same computer hardware concurrently. It's mature and widely used in many cloud environments.

Use Case: Xen is often used in cloud environments, such as Amazon Web Services. It is also used in high-performance computing environments due to its near-native performance.

6. Proxmox

Proxmox VE (Virtual Environment) is an open-source server virtualization management platform. It supports KVM-based virtual machines and LXC for container-based virtualization. It includes a web console and command-line tools, and provides a REST API for third-party tools.

Use Case: Proxmox is typically used as a solution for small to medium-size businesses due to its easy-to-use web interface that can manage multiple virtual machines, containerized workloads, and software-defined storage.

In terms of their relationship, these tools and platforms can often be used together. For example, KVM and QEMU are often used together to provide full virtualization on a Linux system. libvirt can be used to manage both KVM and QEMU, and virt-manager can provide a graphical interface to libvirt. Proxmox, on the other hand, is a complete virtualization management solution that uses KVM for full virtualization. Xen is a standalone type-1 hypervisor that doesn't require KVM or QEMU but can be managed by libvirt.

Re:We Replaced Firecracker with QEMU by doliveira on Monday July 10, 2023 @12:27PM (#36668677)
Honestly, when we're discussing with humans we want the humans' response. Anyone of us can ask ChatGPT, I don't know why people keep pasting its answers
Re:We Replaced Firecracker with QEMU by solarpunk on Monday July 10, 2023 @01:20PM (#36669554)
people keep doing it because it lets them farm upvotes.
Re:We Replaced Firecracker with QEMU by bonzini on Monday July 10, 2023 @12:01PM (#36668291)
Pretty good I must say. AWS however it's using KVM only these days!
Re:We Replaced Firecracker with QEMU by orangepurple on Monday July 10, 2023 @12:34PM (#36668799)
What is AWS Nitro then?
Re:We Replaced Firecracker with QEMU by bonzini on Monday July 10, 2023 @01:07PM (#36669308)
Nitro is the custom hardware that runs in AWS data centers (or in outposts). You can use it bare metal, or if you use VM instances it will run Linux (plus KVM) and a custom user space component that replaces QEMU.
Re:We Replaced Firecracker with QEMU by theossuary on Monday July 10, 2023 @11:53AM (#36668204)
Damn, outdone by ChatGPT
Re:We Replaced Firecracker with QEMU by theossuary on Monday July 10, 2023 @11:51AM (#36668165)
KVM is a Linux kernel implementation of the cpu extensions to accelerate vms to near bare metal speeds.

Qemu is a user space system emulator. It can emulate in software different architectures like ARM, x86, etc. It can also emulate drivers, networking, disks, etc. Is called via the command line.

The reason you'll see Qemu/KVM a lot is because Qemu is the emulator, the things actually running the VM. And it utilizes KVM to speed up the VM when the host architecture matches the VM's.

Libvirt is an XML based API on top of Qemu. It allows you to define networks, VMs (it calls them domains), and much more as XML through libvirtd.

Virsh is a CLI tool to manage libvirtd. Virt-manager is a GUI to do the same.

Proxmox is debian under the hood with Qemu/KVM running VMs. It provides a robust web UI and easy clustering capabilities. Along with nice to haves like easy management of disks, ceph, etc. You can also manage ceph through an API with terraform.

Xen is an alternative hypervisor (like esxi). Instead of running on top of Linux, Xen has it's own microkernel. This means less flexibility (there's no Linux body running things), but also simpler to manage and less attack surface. I haven't played much with xen though, KVM is kind of the defacto, but iirc AWS uses a modified Xen.

Re:We Replaced Firecracker with QEMU by adql on Monday July 10, 2023 @12:20PM (#36668560)
>Xen is an alternative hypervisor (like esxi). Instead of running on top of Linux, Xen has it's own microkernel. This means less flexibility (there's no Linux body running things), but also simpler to manage and less attack surface. I haven't played much with xen though, KVM is kind of the defacto, but iirc AWS uses a modified Xen.

If you actually played with Xen you'd know it's not actually easier to manage. And increased security claims are dubious at best, as same thing that would be attacked (dom0 managing the whole thing and running linux) have direct unfettered access to xen microkernel. There is reason many sites migrated away from Xen to KVM. Also many Xen drivers de facto run part Linux dom0 instance so you don't even get that isolation.

We ran Xen for few years, as KVM at first was still not as refined and Xen was first to mature market, and it was just million little annoying things.

KVM offers far simple and straightforward management. A VM is just a process. You can look at its CPU usage via normal tools. No magic. No driver problems.

Re:We Replaced Firecracker with QEMU by monocasa on Monday July 10, 2023 @01:27PM (#36669679)
> I haven't played much with xen though, KVM is kind of the defacto, but iirc AWS uses a modified Xen.

AWS switched to KVM, and even a lot of AWS systems that report themselves as Xen are running as KVM with a compat shim to say it's Xen.

Re:We Replaced Firecracker with QEMU by cthalupa on Monday July 10, 2023 @01:56PM (#36670289)
> Xen is an alternative hypervisor (like esxi). Instead of running on top of Linux, Xen has it's own microkernel. This means less flexibility (there's no Linux body running things), but also simpler to manage and less attack surface.

You're going to need dom0 (a "control domain") on any Xen host. Gotta have something running xl and the rest of the toolstack for managing it. dom0less technically exists but the drawbacks mean it's not really usable by most people in a production situation.