Re:GPU Embedding with GGML
by waldfee on Monday October 16, 2023 @10:38AM
(#37900450)
i am only dabbling in this space myself, so can't answer everything. all the formats i mentioned are for a quantized version of the original model. basically a lower resolution version, with the associated precision loss. e.g. original model weights are in f16, the gptq version is in int4. a big difference in size but often an acceptable loss of quality. using quants is basically a tradeoff between quality and "can i run it?".
examples of original models are llama(2), mistral, xwin. they are not directly related to any quantized versions. quants are mostly done by third parties (e.g. thebloke[1]).
using a full model for inference requires pretty beefy hardware. most inference on consumer hardware is done with quantized versions for that reason.
examples of original models are llama(2), mistral, xwin. they are not directly related to any quantized versions. quants are mostly done by third parties (e.g. thebloke[1]).
using a full model for inference requires pretty beefy hardware. most inference on consumer hardware is done with quantized versions for that reason.
[1] https://huggingface.co/TheBloke