News for hackers. Stuff that matters.
Re:QuIP#: 2-bit Quantization for LLMs by p1esk on Saturday December 09, 2023 @10:19AM (#38582569)
This is for GPUs, not CPUs. GPUs do have lower precision ALUs to do math on fewer bits. Though not 2 bits - I believe there’s support for 1, 4 and 8 bit computation in modern Nvidia cards.

But even without such support there’s a benefit of model size compression so that bigger models can fit in GPU memory, eliminating costly CPU/GPU data transfers.

There are no comments on this comment yet.