Re:A comprehensive guide to running Llama 2 locally
by shortrounddev2 on Wednesday July 26, 2023 @09:50AM
(#36877690)
To be perfectly honest, I know absolutely nothing about AI or Llama; I'm just a Windows C++ programmer so I wanted to provide cmake instructions for Windows, sorry. The .bin file is what I got from the OP's link
[ Parent
| on HN ]
Re:A comprehensive guide to running Llama 2 locally
by bed147373429 on Wednesday July 26, 2023 @11:15AM
(#36879100)
It's ok, I just followed your instructions and with that model is works well. But are you sure that this uses CUDA? My CPU utilization is at 50% while my GPU utilization is at 1% while the output is being generated..
Re:A comprehensive guide to running Llama 2 locally
by shortrounddev2 on Wednesday July 26, 2023 @11:28AM
(#36879358)
The cmake build prints that it finds cuda when I run the cmakelists (prints the location of cuda headers), however I dont see any noticeable difference between cpu-only and cuda builds. So if its not working then maybe there a CLI option thats required, or maybe cuda support is broken on windows