News for hackers. Stuff that matters.

A comment on Microsoft BitNet: inference framework for 1-bit LLMs. Parent comment

Re:Microsoft BitNet: inference framework for 1-bit LLMs by throwaway314155 on Friday October 18, 2024 @12:33PM (#41880994)
Probably because it has relatively painless GPU sharing with pass through. As far as I know that sort of feature requires a hypervisor-level VM, which is not something you get with VirtualBox.
[ Parent | on HN ]
Re:Microsoft BitNet: inference framework for 1-bit LLMs by diggan on Friday October 18, 2024 @12:43PM (#41881111)
Someone correct me if I'm wrong, but I think you can use a KVM or QEMU backend for VirtualBox and that way get GPU pass-through. Probably not out of the box though.
[ Parent | on HN ]
Re:Microsoft BitNet: inference framework for 1-bit LLMs by Datagenerator on Friday October 18, 2024 @05:31PM (#41883686)
Close Windows, many doors open
[ Parent | on HN ]
Re:Microsoft BitNet: inference framework for 1-bit LLMs by EgoIncarnate on Friday October 18, 2024 @09:23PM (#41884904)
The WSL2 GPU passthrough is more like a virtual GPU than KVM style device passthrough. I believe it's effectively a device specific linux userland driver to device specific windows kernel driver with a linux kernel shim bridging the too. If I recall correctly, the linux userland drivers are actually provided by the windows driver.
[ Parent | on HN ]