News for hackers. Stuff that matters.

A comment on State-of-the-art open-source chatbot, Vicuna-13B, just released model weights. Parent comment

Re:State-of-the-art open-source chatbot, Vicuna-13B, just re... by andai on Wednesday April 05, 2023 @03:38AM (#35450649)
Vicuna's GitHub says that applying the delta takes 60GB of CPU RAM? Is that what you meant by large swap file?

On that note, why is any RAM needed? Can't the files be loaded and diffed chunk by chunk?

[ Parent | on HN ]
Re:State-of-the-art open-source chatbot, Vicuna-13B, just re... by muyuu on Wednesday April 05, 2023 @06:47AM (#35452012)
btw I got 4bit quantized Vicuna working in my 16GB laptop and the results seem very good, perhaps the best I got running locally so far
[ Parent | on HN ]
Re:State-of-the-art open-source chatbot, Vicuna-13B, just re... by andai on Friday April 07, 2023 @01:21PM (#35484227)
Did you have to diff LLaMA? Did you use EasyLM?
[ Parent | on HN ]
Re:State-of-the-art open-source chatbot, Vicuna-13B, just re... by muyuu on Friday April 07, 2023 @05:54PM (#35487493)
I found it ready-made for download, here https://huggingface.co/eachadea/ggml-vicuna-13b-4bit
[ Parent | on HN ]