Re:70B Llama 2 at 35tokens/second on 4090
by brucethemoose2 on Wednesday September 13, 2023 @05:13PM
(#37501805)
That is it ^. Search for the parameter count you want (like "70B" or "13B") and the format you desire ("GPTQ" or "GGUF")
Some users are really terrible about labeling their model cards though, and some models may not have any GPTQ/GGUF files (meaning you have to convert them yourself).
Some users are really terrible about labeling their model cards though, and some models may not have any GPTQ/GGUF files (meaning you have to convert them yourself).