Re:Microsoft BitNet: 100B Param 1-Bit model for local CPUs
by buo on Wednesday March 11, 2026 @01:06PM
(#47338231)
Do you know a good reference to learn more about this (quantizing weigths to 1.58 bits, and trit arithmetic)?
Re:Microsoft BitNet: 100B Param 1-Bit model for local CPUs
by fc417fc802 on Wednesday March 11, 2026 @04:50PM
(#47341520)
There's lots of literature on quantizing weights (including trits and binary) going back 15+ years. Nothing to hand right now but it's all on arxiv.
The relevant trit arithmetic should be on display in the linked repo (I haven't checked). Or try working it out for the uncompressed 2 bit form with a pen and paper. It's quite trivial. Try starting with a couple bitfields (inputs and weights), a couple masks, and see if you can figure it out without any help.
Interestingly, a trit x float multiplier is cheaper than a trit x integer multiplier in hardware if you're willing to ignore things like NaNs.
0 and 1 are trivial, just a mux for identity and zero. But because floats are sign-magnitude, multiply by -1 is just an inverter for the sign bit, where as for integers you need a bitwise inverter and full incrermenter.