the key is that with Virtlet you can use you windows VM as other pods in replicasets, daemonsets, statefulsets, what can not be done using CRD offered by kubevirt. In the same time, assuming that your windows image supports cloud init (e.g. using http://cloudbase-init.readthedocs.io ) - you can setup it during the boot, configuring users, passwords, running arbitrary scripts - you have full control about what will be done during cloud init phase using only pod annotations (you don't need prepared specific image with predefined content as in KubeVirt case). In case of Virtlet you have a possibility to merge data from different sources (config maps, secrets, annotations) and use the result in output cloud init image.

At the end - KubeVirt provides you a way to run a VM as a custom resource in an app on k8s (virt-launcher), while Virtlet provides a way to run VM as first class cluster citizen, as a pod, with the same interfaces as for other pods (so you can use kubectl logs, kubectl attach -it, as with other "normal" (docker image based) pods).

ok, with windows vm you will see in kubectl logs only cloud init logs as thats the only part which is using serial console to output something :P

with other types of vm images (based on linux, bsd or unikernels) you would probably appreciate more possibility to use kubectl attach -it, or kubectl logs ;)

btw. both solutions have they strong points - virtlet is a bit closer to k8s interfaces, while kubevirt provides closer to libvirt options (e.g. TAL on https://twitter.com/dummdida/status/992037913352392705 )

Regarding CRDs and core types like statefulsets: We could reuse them with CRDs too. We explicitly decided against it to give the user and the internal components the chance to work with proper abstractions. We for instance have a real dedicated REST-API, fully integrated into k8s, including websocket endpoints for console, vnc, ...: https://www.kubevirt.io/api-reference/master/operations.html. Regarding to networking integration and cluster administration our vms are transparent to the cluster to ensure seamless integration on the pod level.

We don't have every controller type available in KubeVirt, but we have for instance a VirtualMachineReplicaSet and an OfflineVirtualMachine (soon renamed to StatefulVirtualMachine): https://www.kubevirt.io/user-guide/#/workloads/controllers/R.... A VirtualMachineStatefulSet will definitely be added. Others will be added on demand.

Regarding disk: I am not sure why you think that there are special configurations necessary on the images to run a vm on KubeVirt. Maybe you can explain that in more detail. It is one of the core use-cases that you simply start your already existing vm without modifications on the images in KubeVirt. We support different volume types, including a RegistryDisk. See https://www.kubevirt.io/user-guide/#/workloads/virtual-machi... for further details.

And of course we support cloud-init: https://www.kubevirt.io/user-guide/#/workloads/virtual-machi....

It is true that we do not integrate as far with "kubectl" like virtlet does. We have our own "virtctl" tool which provides virtualization related commands (e.g. "virtctl console", "virtctl vnc", ...). Some of them replacing a missing kubectl pieces, some of them adding virtualization specific commands. Having the deep integration is indeed nice. I just want to add that solely depending on "kubectl" is a two-edged sword. For some vms things work, for others not. There is a good chance that people have to modify their images to properly integrate with "kubectl" in the mentioned ways. That is exactly what we did not want.

Personally I think that there are differences between pods and virtual machines (arguably varying depending on what you do with your workloads). Integrating where things are equal and extending otherwise is a good thing. It allows very natural interactions.

I would be careful with saying that one or the other project allows running vms more as "first class citizen" in the cluster than the other. We could probably argue about that forever.

Yes, you can use KubeVirt for Windows. In some cases, it may be even better than Virtlet for that purpose. But if you want a real Kubernetes Deployment / StatefulSet / DaemonSet / etc. built out of VMs (examples being unikernels, nested Kubernetes cluster for testing etc.) Virtlet may be a better choice.

True - Today it does not directly fit into existing workload controllers (Deployment, ReplicaSet etc) but SIG Api Machinery actually saw that others would also like to see CRD support in templates of controllers.

If this lands, then KubeVirt would get that support "for free" (tm)

Regarding the nested cluster: One of the main reasons for having the VirtualMachineReplicaSet in KubeVirt is to allow that use-case. They work pretty nice in combination with CloudInit and e.g. kubeadm to dynamically scale the cluster.