News for hackers. Stuff that matters.
Re:Run LLaMA and Alpaca on your computer by mastazi on Wednesday April 05, 2023 @06:09PM (#35461084)
How so?

how is this:

    npx dalai llama 7B
    npx dalai serve
harder than:

    # build this repo
    git clone https://github.com/ggerganov/llama.cpp
    cd llama.cpp
    make

    #For Windows and CMake, use the following command instead:
    cd <path_to_llama_folder>
    mkdir build
    cd build
    cmake ..
    cmake --build . --config Release

    # obtain the original LLaMA model weights and place them in ./models
    ls ./models
    65B 30B 13B 7B tokenizer_checklist.chk tokenizer.model

    # install Python dependencies
    python3 -m pip install torch numpy sentencepiece

    # convert the 7B model to ggml FP16 format
    python3 convert-pth-to-ggml.py models/7B/ 1

    # quantize the model to 4-bits (using method 2 = q4_0)
    ./quantize ./models/7B/ggml-model-f16.bin ./models/7B/ggml-model-q4_0.bin 2

    # run the inference
    ./main -m ./models/7B/ggml-model-q4_0.bin -n 128
Re:Run LLaMA and Alpaca on your computer by hrpnk on Wednesday April 05, 2023 @07:12PM (#35461786)
dalai does a lot of magic in the background. I had to run the package from source to debug the error messages. When run in on a Mac with zsh, it still runs bash under the hood to setup the python venv. Took me an hour to debug that as bash did not have poetry and pyenv configured to use <3.11 python since I never use bash. Yet, sentencepiece does not have a 3.11 wheel available, thus running dalai fails.

Today, just 2d after running dalai successfully, it refuses to start and just hangs in there. Unclear why.

Re:Run LLaMA and Alpaca on your computer by MintPaw on Thursday April 06, 2023 @12:31AM (#35464154)
llama.cpp and alpaca.cpp both ship binary releases. Dalai does not seem to.
Re:Run LLaMA and Alpaca on your computer by vorticalbox on Thursday April 06, 2023 @06:38AM (#35466570)
for one the script after cloning llama.cpp repo then calls `exit()` which then drops me out of my fish shell.
Re:Run LLaMA and Alpaca on your computer by msm_ on Thursday April 06, 2023 @05:39PM (#35474899)
The first version won't work on my linux distribution (nixos). The second one also won't work, but I know how to make it work (C++ and Python without any additional abstraction layers are easy).