70B Llama 2 at 35tokens/second on 4090

Re:70B Llama 2 at 35tokens/second on 4090 by acheong08 on Wednesday September 13, 2023 @01:11PM (#37499045)

I’ve been a away from LLMs for a while. Is there something like CivitAI for LLMs to find models for certain niches?

[ Parent | on HN ]

Re:70B Llama 2 at 35tokens/second on 4090 by brucethemoose2 on Wednesday September 13, 2023 @05:13PM (#37501805)

That is it ^. Search for the parameter count you want (like "70B" or "13B") and the format you desire ("GPTQ" or "GGUF")

Some users are really terrible about labeling their model cards though, and some models may not have any GPTQ/GGUF files (meaning you have to convert them yourself).

[ Parent | on HN ]