Re:You can now train a 70B language model at home
by mmoskal on Saturday March 09, 2024 @12:36AM
(#39649648)
The bitnet paper was showing worse results than fp16 transformer with the same parameter count. The shocking result in the 1.58b paper (same group) is no quality loss compared to fp16.
Re:You can now train a 70B language model at home
by paul_mk1 on Saturday March 09, 2024 @05:22PM
(#39655176)
Goes back before then. This got popularized by BinaryConnect in 2015, and groups were training binary networks as early as 2011.
You are probably referring to XNOR net, and the novel piece there was also using binary activations (which bitnet is not).
So as far as I can tell, bitnet is basically BinaryConnect applied to LLMs.
A side point, but this "recent" encoding goes back to a 2017 paper from the Allen Institute. These days a seven year old paper is ancient.
They went further and showed you could could get away with binary, you don't even need trinary!