A comprehensive guide to running Llama 2 locally

Re:A comprehensive guide to running Llama 2 locally by shortrounddev2 on Tuesday July 25, 2023 @08:46PM (#36871730)

For my fellow Windows shills, here's how you actually build it on windows:

Before steps:

1. (For Nvidia GPU users) Install cuda toolkit https://developer.nvidia.com/cuda-downloads

2. Download the model somewhere: https://huggingface.co/TheBloke/Llama-2-13B-chat-GGML/resolv...

In Windows Terminal with Powershell:

    git clone https://github.com/ggerganov/llama.cpp
    cd llama.cpp
    mkdir build
    cd build
    cmake .. -DLLAMA_CUBLAS=ON
    cmake --build . --config Release
    cd bin/Release
    mkdir models
    mv Folder\Where\You\Downloaded\The\Model .\models
    .\main.exe -m .\models\llama-2-13b-chat.ggmlv3.q4_0.bin --color -p "Hello, how are you, llama?" 2> $null

`-DLLAMA_CUBLAS` uses cuda

`2> $null` is to direct the debug messages printed to stderr to a null file so they don't spam your terminal

Here's a powershell function you can put in your $PSPROFILE so that you can just run prompts with `llama "prompt goes here"`:

    function llama {
        .\main.exe -m .\models\llama-2-13b-chat.ggmlv3.q4_0.bin -p $args 2> $null
    }

adjust your paths as necessary. It has a tendency to talk to itself.

[ Parent | on HN ]

Re:A comprehensive guide to running Llama 2 locally by jmorgan on Tuesday July 25, 2023 @11:00PM (#36872616)

> adjust your paths as necessary. It has a tendency to talk to itself.

This is always a fun surprise. What I've seen help, especially with chat models, is to use a prompt template. Some tools (e.g. https://ollama.ai/ – mentioned in the article) use a default, model-specific prompt template when you run the model. This is easier for users since they can just input your chat messages and get answers. The hard part is every model is trained (and behaves) differently.

With llama.cpp you'd need wrap your prompt text with the right template. For llama 2, the facebook developers' generation code wraps the system prompt and user prompts in specific tags (<<SYS>>{system prompt} and [INST]{user prompt}[/INST]) respectively): https://github.com/facebookresearch/llama/blob/main/llama/ge....

Worth noting that customizing prompt templates can be fun – you don't have to use "prescribed" one – I've had a model generate a conversation between a few characters that talk to each other for example – it's pretty entertaining!

[ Parent | on HN ]

Re:A comprehensive guide to running Llama 2 locally by lhl on Wednesday July 26, 2023 @04:01AM (#36874448)

As a followup on LLama2's prompting, here's a good thread with some more details: https://www.reddit.com/r/LocalLLaMA/comments/155po2p/get_lla... (see also: https://www.reddit.com/r/LocalLLaMA/comments/1561vn5/here_is... - it's a bit complex and people are still wrapping their heads around it)

Note, it is possible to reinject <<SYS>> prompts during the conversation to keep the LLM on target: https://twitter.com/overlordayn/status/1681631554672513025 (but obviously this should be the first thing you filter and track if you are running an LLM that is accessible to end-users - rebuff is a lib w/ some good ideas to start with)

With new fine tunes coming out, it's worth noting again that different datasets/models all use slightly different prompt formats (many are using multiple datasets these days, so it's hard to say just how much it matters now): https://www.reddit.com/r/LocalLLaMA/comments/13lwwux/comment...

Ideally, fine tunes for LLama2 would regularize all datasets to the official tokens/format and inference interfaces could standardize too (or there could be some metadata collected for what model uses what, but some of the low hanging fruit that's out for the future I guess). One other thing to keep in mind is that all the benchmarks/leaderboards don't use instruct formatting at all, and don't really represent capabilities for a model. IMO, ELO-style rankings against models for specific tasks would probably be more representative.

[ Parent | on HN ]

Re:A comprehensive guide to running Llama 2 locally by bed147373429 on Wednesday July 26, 2023 @08:35AM (#36876601)

I have a question: Last week I downloaded llama-7b-chat from meta's github directly (https://github.com/facebookresearch/llama) using the URL they sent via e-mail. As a result, I now have the model as consolidated.00.pth.

Your commands assume the model is a .bin file (so I guess there must be a way to convert the pytorch model .pth to the .bin file). How can I do this and what is the difference between the two models?

The facebook repo provides commands for using the models, these commands don't work on my windows machine: "NOTE: Redirects are currently not supported in Windows or MacOs. [W ..\torch\csrc\distributed\c10d\socket.cpp:601] [c10d] The client socket has failed to connect to ...."

The facebook repo does not describe which OS you are supposed to use, so I assumed it would work on Windows too. But then if this can work why would anyone need the ggerganov llama code? I am new to all of this and easily confused, so any help is appreciated

[ Parent | on HN ]

Re:A comprehensive guide to running Llama 2 locally by shortrounddev2 on Wednesday July 26, 2023 @09:50AM (#36877690)

To be perfectly honest, I know absolutely nothing about AI or Llama; I'm just a Windows C++ programmer so I wanted to provide cmake instructions for Windows, sorry. The .bin file is what I got from the OP's link

[ Parent | on HN ]

Re:A comprehensive guide to running Llama 2 locally by bed147373429 on Wednesday July 26, 2023 @11:15AM (#36879100)

It's ok, I just followed your instructions and with that model is works well. But are you sure that this uses CUDA? My CPU utilization is at 50% while my GPU utilization is at 1% while the output is being generated..

[ Parent | on HN ]

Re:A comprehensive guide to running Llama 2 locally by shortrounddev2 on Wednesday July 26, 2023 @11:28AM (#36879358)

The cmake build prints that it finds cuda when I run the cmakelists (prints the location of cuda headers), however I dont see any noticeable difference between cpu-only and cuda builds. So if its not working then maybe there a CLI option thats required, or maybe cuda support is broken on windows

[ Parent | on HN ]

Re:A comprehensive guide to running Llama 2 locally by pdntspa on Wednesday July 26, 2023 @10:07AM (#36877970)

llama.cpp needs the files to be in ggml format, there is a command string you can run to convert one from the other (as well as perform quantization). Or just download the GGML version

https://www.reddit.com/r/LocalLLaMA/wiki/models#wiki_llama_2...

[ Parent | on HN ]

Re:A comprehensive guide to running Llama 2 locally by spike_protein on Saturday July 29, 2023 @01:11PM (#36922307)

try *cd llama.cpp && python convert-pth-to-ggml.py models/7B/ 1*

[ Parent | on HN ]

Re:A comprehensive guide to running Llama 2 locally by appel on Wednesday July 26, 2023 @11:13AM (#36879072)

Thanks for sharing these instructions, I really appreciate it.

[ Parent | on HN ]

Re:A comprehensive guide to running Llama 2 locally by bed147373429 on Wednesday July 26, 2023 @04:42PM (#36884699)

I also wrote a small Python script that you can also use if you do not want to use Powershell to talk to your Llama model: https://downioads.github.io/posts/llama-python/

[ Parent | on HN ]

Re:A comprehensive guide to running Llama 2 locally by arturogatti on Thursday August 03, 2023 @06:36AM (#36983902)

Gives you an undetectable virus. Also requires 1800 gigs of ram

[ Parent | on HN ]

Re:A comprehensive guide to running Llama 2 locally by shanks512 on Wednesday July 26, 2023 @04:49PM (#36884789)

How much storage and memory does this use?

[ Parent | on HN ]

Re:A comprehensive guide to running Llama 2 locally by arturogatti on Thursday August 03, 2023 @06:40AM (#36983929)

Nobody here gave you correct info. You must have 14,000 gb of ram to run a real dataset not monitored by the people with brains

[ Parent | on HN ]

Re:A comprehensive guide to running Llama 2 locally by vsnf on Wednesday July 26, 2023 @05:39PM (#36885448)

> adjust your paths as necessary.

What does this mean, exactly, and how does it prevent llama from carrying on a (rather pleasant, actually) conversation with itself?

[ Parent | on HN ]

Re:A comprehensive guide to running Llama 2 locally by shortrounddev2 on Wednesday July 26, 2023 @10:43PM (#36888253)

I just mean where I have a relative path, set your own paths to what they would be on your machine. The subsequent comment is unrelated

[ Parent | on HN ]