Every story and comment from the Hacker News API, in a layout borrowed from 2001.
A comment on 2-bit and 4-bit versions of Mixtral.
The 2-bit version can run on a 24GB Titan RTX.
In terms of perplexity scores on the wikitext2 dataset, the results are as follows: Mixtral: 26GB / 3.79 Llama2-70B: 26.37GB / 4.13
There are no comments on this comment yet.
The 2-bit version can run on a 24GB Titan RTX.
In terms of perplexity scores on the wikitext2 dataset, the results are as follows: Mixtral: 26GB / 3.79 Llama2-70B: 26.37GB / 4.13