- bitsandbytes was always used for on the fly 8 bit quant, just like its being used for 4-bit now. - llama.cpp (and derivatives) quantize ahead of time, but its not resource intense. - mlc llm (vulkan/metal llm inference via tvm) do require lots of ram for their quantization
LLM8 was introduced before https://arxiv.org/abs/2208.07339 of the same first and last author as QLora & is still what can be used in Huggingface Transformers with the `load_in_8bits` parameter.
The idea was just to quantize all weights to 8 bits except a few outliers, which are kept in original precision. This scheme kept the computations extremely accurate, and was really fast to do.
I haven't read the new paper, but I assume they came up with a more advanced fast distributional setup.
If you're an enthusiast with 10 models downloaded, do you want that taking up 500GB or 150GB? Do you want to need 64GB of RAM to load a model, or just 16GB?
That's the main reason for the popularity of pre-quantization.
So you still need enough system RAM (or RAM+Swap) to load the unquantized model.
You could absolutely do something streaming, or mmap the weights instead of loading them into system RAM. Just the default interfaces don't.
The colab notebook shows an example of loading the vanilla, unquantized model "decapoda-research/llama-7b-hf", using the flag "load_in_4bit" to load it as 4bits.
When... when did this become possible? My understanding, from playing with these models daily for the past few months, is that quantization of LLaMA-based models is done via this: https://github.com/qwopqwop200/GPTQ-for-LLaMa
And performing the quantization step is memory and time expensive. Which is why some kind people with large resources are performing the quantization, and then uploading those quantized models, such as this one: https://huggingface.co/TheBloke/wizard-vicuna-13B-GPTQ
But now I'm seeing that, as of recently, the transformers library is capable of loading models in 4bits simply by passing this flag?
Is this a free lunch? Is GPTQ-for-LLaMA no longer needed anymore? Or is this still not as good, in terms of inference quality, as the GPTQ-quantized models?