Klipper3
Re:The Era of 1-bit LLMs: ternary parameters for cost-effective computing
on Wednesday February 28, 2024 @05:19PM (#39544211)
INT2 ternary is equivalent to INT1 + binary mask. Nvidia supprted INT1 matrix multiply in RTX20 and RTX30 generations, nobody used it, so they removed INT1 support from RTX40 generation.
Re:The Era of 1-bit LLMs: ternary parameters for cost-effective computing
on Wednesday February 28, 2024 @07:58AM (#39537262)
The theoretical capacity of a binary network is 69% of the capacity of a full-weight network, so it makes sense that LLM would converge to 1-bit networks in the long term.
It's nice to finally see practical networks reach the theoretical limits found in the statistical mechanics of Ising models. A good pointer to efficient 1-bit training, from the statistical mechanics point of view, is here: