Re:Microsoft BitNet: inference framework for 1-bit LLMs
by newfocogi on Friday October 18, 2024 @10:28AM
(#41879780)
I'm enthusiastic about BitNet and the potential of low-bit LLMs - the papers show impressive perplexity scores matching full-precision models while drastically reducing compute and memory requirements. What's puzzling is we're not seeing any major providers announce plans to leverage this for their flagship models, despite the clear efficiency gains that could theoretically enable much larger architectures. I suspect there might be some hidden engineering challenges around specialized hardware requirements or training stability that aren't fully captured in the academic results, but would love insights from anyone closer to production deployment of these techniques.
[ Parent
| on HN ]
Re:Microsoft BitNet: inference framework for 1-bit LLMs
by swfsql on Friday October 18, 2024 @11:16AM
(#41880200)
I think that since training must happen on a non-bitnet architecture, tuning towards bitnet is always a downgrade on it's capabilities, so they're not really interested in it.
But maybe they could be if they'd offer cheaper plans, since it's efficiency is relatively good.
I think the real market for this is for local inference.
Re:Microsoft BitNet: inference framework for 1-bit LLMs
by waynenilsen on Friday October 18, 2024 @11:35AM
(#41880375)
I suppose hardware support would be very helpful, new instructions for bitpacked operations?
Re:Microsoft BitNet: inference framework for 1-bit LLMs
by strangescript on Friday October 18, 2024 @10:44AM
(#41879903)
I find it a little confusing as well. I wonder if its because so many of these companies have went all in on the "traditional" approach that deviating now seems like a big shift?
Re:Microsoft BitNet: inference framework for 1-bit LLMs
by danielmarkbruce on Friday October 18, 2024 @12:55PM
(#41881230)
People are almost certainly working on it. The people who are actually serious and think about things like this are less likely to just spout out "WE ARE BUILDING A CHIP OPTIMIZED FOR 1-BIT" or "WE ARE TRAINING A MODEL USING 1-BIT" etc, before actually being quite sure they can make it work at the required scale. It's still pretty researchy.