On the 1.58bit paper, from what I understand, this requires a total retraining from scratch. Hopefully the researchers will open source their weights :)
On the technicals, weights are encoded in (-1, 0, 1), whilst QLoRA uses a 4bit dynamic mapping of 16 numbers. The only change required would be the torch.matmul(X, W) step, where it'll be torch.bitlinear_matmul(X, W). Before with QLoRA, one has to do torch.matmul(X, dequantize(W)). So one has to implement torch.bitlinear_matmul. The backward is torch.bitlinear_matmul(dY, W.T).
[-1, 0, 1]
[0, 1, -1]
[10, 20, 30] x [1, 1, 0]
Instead of doing 10(-1) + 20(0) + 30(1) + 10(0) + ..., since we know beforehand the weights are simply (-1, 0, 1), we easily flip the sign and do addition, or force the hardware to do addition ie if (-1) do subtraction. If (0) do addition. If (1) do addition.Floating point multiplication does addition of the exponents and multiplying of the mantissa. So just simplifying:
Float16 has E=5, M=10. Ie around 5 + 10^2 space needed = 105.
Bfloat16 has E=8, M=7. So 8 + 7^2 = 57 space.
Float8(143) E=4, M=3. So 4 + 3^2 = 13 space.
1.58(16bit) E=5, M=10. Addition only, so shift E say 5 + 10 addition = 15.
1.58(8bit) E=4, M=3. Addition only, so shift E say 4 + 3 addition = 7.
Obviously I'm simplifying, but with only additions, 1.58 uses say 7 space, whilst FP8 uses 13 space, so in theory 2x more transistors can be crammed, ie 2x more FLOPs than FP8.
So if you train your models with that in mind you're weighs can be reduced to -1,0,1 reducing the space complexity.
I don't think the costs in expressiveness are captured quite yet, but as perplexity doesn't care about correctness, if that is the metric that is important for you it will probably reduce memory requirements for inference.
If you use Stylus (or any similar browser extension), I actually wrote a style to hide points for that very reason, replacing karma and scores with `•••`
This is actually the second time I see someone mentioning this need, so I've made it into a gist and published it to userstyles, but here's it is also since it's pretty short:
@-moz-document domain("news.ycombinator.com") {
/* Hide karma and points on replies */
span.pagetop #karma, span.comhead span.score {
visibility: hidden;
position: relative;
display: inline-block;
height: 10px !important;
overflow: hidden;
}
span.pagetop #karma {
width: 0.8rem !important;
}
span.comhead span.score {
width: 0.8rem !important;
}
span.pagetop #karma::before, span.comhead span.score::before {
content: "•••";
visibility: visible;
overflow: hidden;
opacity: 0.8;
font-family: Helvetica, Arial, sans-serif !important;
}
}
https://gist.github.com/airstrike/62584e6ffb6104791c0ae48a8e...https://userstyles.world/style/15164/hackernews-hide-karma-a...
P.S. I thought one was suppose to spend the HN points on mocking north-americans, shameless self-promotion, unpopular facts, general trolling and complaints about topics existing. I could go on but I haven't the points.
A side point, but this "recent" encoding goes back to a 2017 paper from the Allen Institute. These days a seven year old paper is ancient.
They went further and showed you could could get away with binary, you don't even need trinary!
You are probably referring to XNOR net, and the novel piece there was also using binary activations (which bitnet is not).
So as far as I can tell, bitnet is basically BinaryConnect applied to LLMs.
The irony of making an unnecessary edit like this to virtue signal for implicit social currency by shitting on the explicit form.
If I were a consumer I would be interested in the final price of fine tuning, for example a table with model size, training size, cost of training, and expected loss of quality with this technology.
One obvious question: Can you apply your technology with the recent (-1,0,1) encoding?, I think you will answers that the (-1,0,1) model is not available and you can't try it, but my question is whether once/if that model is available answer.ai will be able to use the same technology that this post to fine tune a big model in two very small GPUs, and then I should ask for a new table with cost/benefits analysis.
Edited: I should add that I find this kind of work very useful for enhancing individual users like me to be able to compete in the applications of LLM market, this is great work and along the lines of the book "from zero to one" (not that I like or dislike the author) to solve the kind of problem that nobody is trying to solve.
Edited: Now that I have a total of 23 points in HN, I will change my password to some random one, just to cure my desire to look for votes, and try to make some work, and again some tomorrow create a new presence in HN.