Microsoft BitNet: inference framework for 1-bit LLMs

Re:Microsoft BitNet: inference framework for 1-bit LLMs by faragon on Friday October 18, 2024 @11:26AM (#41880304)

I'm glad Microsoft uses Bash in the example, instead of their own Windows shells. As a user I would like having something like "Git Bash" for Windows built in the system, as default shell.

[ Parent | on HN ]

Re:Microsoft BitNet: inference framework for 1-bit LLMs by not_a_bot_4sho on Friday October 18, 2024 @12:10PM (#41880730)

WSL is where it's at today. It's not quite what you're asking for, as it is a separate virtual OS, but the integration is so tight that it feels like you're using your favorite shell natively in Windows.

[ Parent | on HN ]

Re:Microsoft BitNet: inference framework for 1-bit LLMs by diggan on Friday October 18, 2024 @12:18PM (#41880840)

> integration is so tight that it feels like you're using your favorite shell natively in Windows

WSL1 certainly felt that way, WSL2 just feels like any other virtualization manager and basically works the same. Not sure why people sings the praise of WSL2, I gave it a serious try for months but there is a seemingly endless list of compatibility issues which I never had with VMWare or VirtualBox, so I just went back to those instead and the experience is the same more or less.

[ Parent | on HN ]

Re:Microsoft BitNet: inference framework for 1-bit LLMs by throwaway314155 on Friday October 18, 2024 @12:33PM (#41880994)

Probably because it has relatively painless GPU sharing with pass through. As far as I know that sort of feature requires a hypervisor-level VM, which is not something you get with VirtualBox.

[ Parent | on HN ]

Re:Microsoft BitNet: inference framework for 1-bit LLMs by diggan on Friday October 18, 2024 @12:43PM (#41881111)

Someone correct me if I'm wrong, but I think you can use a KVM or QEMU backend for VirtualBox and that way get GPU pass-through. Probably not out of the box though.

[ Parent | on HN ]

Re:Microsoft BitNet: inference framework for 1-bit LLMs by Datagenerator on Friday October 18, 2024 @05:31PM (#41883686)

Close Windows, many doors open

[ Parent | on HN ]

Re:Microsoft BitNet: inference framework for 1-bit LLMs by EgoIncarnate on Friday October 18, 2024 @09:23PM (#41884904)

The WSL2 GPU passthrough is more like a virtual GPU than KVM style device passthrough. I believe it's effectively a device specific linux userland driver to device specific windows kernel driver with a linux kernel shim bridging the too. If I recall correctly, the linux userland drivers are actually provided by the windows driver.

[ Parent | on HN ]

Re:Microsoft BitNet: inference framework for 1-bit LLMs by layer8 on Friday October 18, 2024 @04:12PM (#41883028)

Just install Cygwin.

Not sure what you mean by “default shell”. The default shell on Windows is this: https://en.wikipedia.org/wiki/Windows_shell. I don’t suppose you mean booting into Bash. Windows doesn’t have any other notion of a default shell.

[ Parent | on HN ]

Re:Microsoft BitNet: inference framework for 1-bit LLMs by faragon on Friday October 18, 2024 @06:10PM (#41883960)

I used Cygwin for more than a decade. I prefer Git Bash (msys-based).

[ Parent | on HN ]

Re:Microsoft BitNet: inference framework for 1-bit LLMs by hhejrkrn on Friday October 18, 2024 @07:46PM (#41884494)

I thought msysgit was also cygein based

[ Parent | on HN ]

Re:Microsoft BitNet: inference framework for 1-bit LLMs by aithrowawaycomm on Saturday October 19, 2024 @03:13AM (#41886203)

They are using Windows shells, just not on Macs. This is what the caption for the video says:

> A demo of bitnet.cpp running a BitNet b1.58 3B model on Apple M2:

And this is in their Windows build instructions:

> Important! If you are using Windows, please remember to always use a Developer Command Prompt / PowerShell for VS2022 for the following commands

[ Parent | on HN ]