Dalai: Automatically install, run, and play with LLaMA on your computer

Re:Dalai: Automatically install, run, and play with LLaMA on... by thuttinger on Sunday March 12, 2023 @07:28PM (#35128088)

Works great! However, i had Python 3.11 set up as default python3 in path, and since there is no wheel for torch for 3.11 yet, the script failed. With 3.10 it worked flawlessly.

Small improvement: the node script could check if the model files are already present at the download location and not download them again in this case.

[ Parent | on HN ]

Re:Dalai: Automatically install, run, and play with LLaMA on... by hbbio on Sunday March 12, 2023 @08:44PM (#35128945)

Happened to me as well. Apparently, you can just run:

   python3.10 convert-pth-to-ggml.py models/7B 1
   ./quantize ./models/7B/ggml-model-f16.bin ./models/7B/ggml-model-q4_0.bin 2

And then play with:

    ./main -m ./models/7B/ggml-model-q4_0.bin -t 8 -n 128 -p "..."

[ Parent | on HN ]