News for hackers. Stuff that matters.

A comment on 70B Llama 2 at 35tokens/second on 4090. Parent comment

Re:70B Llama 2 at 35tokens/second on 4090 by brucethemoose2 on Wednesday September 13, 2023 @04:20AM (#37493736)
Some 70B finetunes excel at certain niches, like specific non-English languages, roleplay, fictional writing, or topics GPT4 would refuse to discuss. Some of these are hard to evaluate, but try out (for instance) MythoMax for fiction writing, Airoboros for "uncensored" general use, and Samantha for therapist style chat.

https://huggingface.co/models?sort=modified&search=70b

[ Parent | on HN ]
Re:70B Llama 2 at 35tokens/second on 4090 by acheong08 on Wednesday September 13, 2023 @01:11PM (#37499045)
I’ve been a away from LLMs for a while. Is there something like CivitAI for LLMs to find models for certain niches?
[ Parent | on HN ]
Re:70B Llama 2 at 35tokens/second on 4090 by brucethemoose2 on Wednesday September 13, 2023 @05:13PM (#37501805)
That is it ^. Search for the parameter count you want (like "70B" or "13B") and the format you desire ("GPTQ" or "GGUF")

Some users are really terrible about labeling their model cards though, and some models may not have any GPTQ/GGUF files (meaning you have to convert them yourself).

[ Parent | on HN ]