News for hackers. Stuff that matters.

A comment on State-of-the-art open-source chatbot, Vicuna-13B, just released model weights. Parent comment

Re:State-of-the-art open-source chatbot, Vicuna-13B, just re... by muyuu on Tuesday April 04, 2023 @06:47AM (#35437602)
consumer hardware is a bit vague of a limitation, which I guess it's partly why people are not tracking precisely what runs on what very closely

these could be useful:

https://nixified.ai

https://github.com/Crataco/ai-guide/blob/main/guide/models.m... -> https://old.reddit.com/user/Crataco/comments/zuowi9/opensour...

https://github.com/cocktailpeanut/dalai

the 4-bit quantized version of LLaMA 13B runs on my laptop without a dedicated GPU and I guess the same would apply to quantized vicuna 13B but I haven't tried that yet (converted as in this link but for 13B instead of 7B https://github.com/ggerganov/llama.cpp#usage )

GPT4All Lora's also works, perhaps the most compelling results I've got yet in my local computer - I have to try quantized Vicuna to see how that one goes, but processing the files to get a 4bit quantized version will take many hours so I'm a bit hesitant

PS: converting 13B Llama took my laptop's i7 around 20 hours and required a large swap file on top of its 16GB of RAM

feel free to answer back if you're trying any of these things this week (later I might lose track)

[ Parent | on HN ]
Re:State-of-the-art open-source chatbot, Vicuna-13B, just re... by andai on Wednesday April 05, 2023 @03:38AM (#35450649)
Vicuna's GitHub says that applying the delta takes 60GB of CPU RAM? Is that what you meant by large swap file?

On that note, why is any RAM needed? Can't the files be loaded and diffed chunk by chunk?

[ Parent | on HN ]
Re:State-of-the-art open-source chatbot, Vicuna-13B, just re... by muyuu on Wednesday April 05, 2023 @06:47AM (#35452012)
btw I got 4bit quantized Vicuna working in my 16GB laptop and the results seem very good, perhaps the best I got running locally so far
[ Parent | on HN ]
Re:State-of-the-art open-source chatbot, Vicuna-13B, just re... by andai on Friday April 07, 2023 @01:21PM (#35484227)
Did you have to diff LLaMA? Did you use EasyLM?
[ Parent | on HN ]
Re:State-of-the-art open-source chatbot, Vicuna-13B, just re... by muyuu on Friday April 07, 2023 @05:54PM (#35487493)
I found it ready-made for download, here https://huggingface.co/eachadea/ggml-vicuna-13b-4bit
[ Parent | on HN ]