Re:A comprehensive guide to running Llama 2 locally
by shortrounddev2 on Wednesday July 26, 2023 @09:50AM
(#36877690)
To be perfectly honest, I know absolutely nothing about AI or Llama; I'm just a Windows C++ programmer so I wanted to provide cmake instructions for Windows, sorry. The .bin file is what I got from the OP's link
Re:A comprehensive guide to running Llama 2 locally
by bed147373429 on Wednesday July 26, 2023 @11:15AM
(#36879100)
It's ok, I just followed your instructions and with that model is works well. But are you sure that this uses CUDA? My CPU utilization is at 50% while my GPU utilization is at 1% while the output is being generated..
Re:A comprehensive guide to running Llama 2 locally
by shortrounddev2 on Wednesday July 26, 2023 @11:28AM
(#36879358)
The cmake build prints that it finds cuda when I run the cmakelists (prints the location of cuda headers), however I dont see any noticeable difference between cpu-only and cuda builds. So if its not working then maybe there a CLI option thats required, or maybe cuda support is broken on windows
Re:A comprehensive guide to running Llama 2 locally
by pdntspa on Wednesday July 26, 2023 @10:07AM
(#36877970)
llama.cpp needs the files to be in ggml format, there is a command string you can run to convert one from the other (as well as perform quantization). Or just download the GGML version
https://www.reddit.com/r/LocalLLaMA/wiki/models#wiki_llama_2...
Re:A comprehensive guide to running Llama 2 locally
by spike_protein on Saturday July 29, 2023 @01:11PM
(#36922307)
try *cd llama.cpp && python convert-pth-to-ggml.py models/7B/ 1*
Your commands assume the model is a .bin file (so I guess there must be a way to convert the pytorch model .pth to the .bin file). How can I do this and what is the difference between the two models?
The facebook repo provides commands for using the models, these commands don't work on my windows machine: "NOTE: Redirects are currently not supported in Windows or MacOs. [W ..\torch\csrc\distributed\c10d\socket.cpp:601] [c10d] The client socket has failed to connect to ...."
The facebook repo does not describe which OS you are supposed to use, so I assumed it would work on Windows too. But then if this can work why would anyone need the ggerganov llama code? I am new to all of this and easily confused, so any help is appreciated