News for hackers. Stuff that matters.

A comment on Show HN: LlamaGPT – Self-hosted, offline, private AI chatbot, powered by Llama 2. Parent comment

Re:Show HN: LlamaGPT – Self-hosted, offline, private AI chat... by mayankchhabra on Wednesday August 16, 2023 @07:07PM (#37155070)
It's fairly straightforward to add GPU support when running on the host, but LlamaGPT runs inside a Docker container, and that's where it gets a bit challenging.
[ Parent | on HN ]
Re:Show HN: LlamaGPT – Self-hosted, offline, private AI chat... by stavros on Wednesday August 16, 2023 @08:10PM (#37155556)
It shouldn't, nVidia provides a CUDA Docker plugin that lets you expose your GPU to the container, and it works quite well.
[ Parent | on HN ]
Re:Show HN: LlamaGPT – Self-hosted, offline, private AI chat... by dicriseg on Thursday August 17, 2023 @03:26PM (#37166707)
See above if you're interested in that. It does work quite well, even with nested virtualization (WSL2).
[ Parent | on HN ]
Re:Show HN: LlamaGPT – Self-hosted, offline, private AI chat... by stavros on Thursday August 17, 2023 @03:40PM (#37166884)
I am, thanks!
[ Parent | on HN ]