Re:1-Bit LLMs Could Solve AI's Energy Demands
by renonce on Friday May 31, 2024 @10:05AM
(#40535352)
For a 4096x4096 matrix its symmetry group has size 4096!. The Stirling’s approximation gives ln(4096!)=4096ln(4096)-4096 which is about 10~11 bits per 4096 numbers. This is less than 0.003 bit per parameter saved.
[ Parent
| on HN ]
There are no comments on this comment yet.