2-bit and 4-bit versions of Mixtral Posted by ibuildthings on Monday December 11, 2023 @03:09PM from the huggingface.co dept. https://huggingface.co/collections/mobiuslabsgmbh/mixtral-hqq-quantized-models-65776b2edddc2360b0edd451 ( 1 comment | 2 points | on HN )
The 2-bit version can run on a 24GB Titan RTX.
In terms of perplexity scores on the wikitext2 dataset, the results are as follows: Mixtral: 26GB / 3.79 Llama2-70B: 26.37GB / 4.13