News for hackers. Stuff that matters.

Running Llama2 Models Locally

I get approval from meta, then I downloaded all Llama2 models (7B,7B-chat,13B,13B-chat,70B,70B-chat) locally, when I tried to run the script to test the 7B model using the command that is mentioned on Llama github "torchrun --nproc_per_node 1 example_text_completion.py \ --ckpt_dir llama-2-7b/ \ --tokenizer_path tokenizer.model \ --max_seq_len 128 --max_batch_size 4" I got this error "Distributed package doesn't have NCCL built in". My environment : Windows 10, Nvidia GeForce RTX 3090, CUDA 11.8, torch 2.0.1+cu118. I have free enough space and I have everything but I don't know why it's not working. Any help please. I'm blocked in this task for 3 days.
( 2 comments | on HN )
Re:Running Llama2 Models Locally by yy77 on Tuesday August 15, 2023 @04:41AM (#37131582)
why not try huggingface?
Re:Running Llama2 Models Locally by WantonQuantum on Tuesday August 15, 2023 @04:02AM (#37131399)
Hacker News isn’t really a tech support forum. You could try the PyTorch forums.

I found this: https://discuss.pytorch.org/t/runtimeerror-distributed-packa...