Re:A comprehensive guide to running Llama 2 locally
by bed147373429 on Wednesday July 26, 2023 @11:15AM
(#36879100)
It's ok, I just followed your instructions and with that model is works well. But are you sure that this uses CUDA? My CPU utilization is at 50% while my GPU utilization is at 1% while the output is being generated..
[ Parent
| on HN ]
Re:A comprehensive guide to running Llama 2 locally
by shortrounddev2 on Wednesday July 26, 2023 @11:28AM
(#36879358)
The cmake build prints that it finds cuda when I run the cmakelists (prints the location of cuda headers), however I dont see any noticeable difference between cpu-only and cuda builds. So if its not working then maybe there a CLI option thats required, or maybe cuda support is broken on windows