Re:1-Bit LLMs Could Solve AI's Energy Demands
by segmondy on Friday May 31, 2024 @08:22AM
(#40534167)
depends on how far. Q8's are pretty much on par with f16/f32 weights. Q5/Q6's if they drop off barely drop below 2%. folks are running Q2, Q3, etc due to being GPU poor, of course it's going to be terrible. When you want to run a 70B model that's originally 140gb and all you have is 16gb of vram.
[ Parent
| on HN ]
There are no comments on this comment yet.