Re:1-Bit LLMs Could Solve AI's Energy Demands
by renonce on Friday May 31, 2024 @10:05AM
(#40535352)
For a 4096x4096 matrix its symmetry group has size 4096!. The Stirling’s approximation gives ln(4096!)=4096ln(4096)-4096 which is about 10~11 bits per 4096 numbers. This is less than 0.003 bit per parameter saved.
I did a rough calculation, and as the precision of the scalar weights decreases, the information content of the specific network permutation becomes a much higher percentage of its overall size.
Depending on the particular neural network architecture, there may be other symmetries beside the symmetric group that also represent compressible redundancies.