Show HN: LlamaGPT – Self-hosted, offline, private AI chatbot, powered by Llama 2

Re:Show HN: LlamaGPT – Self-hosted, offline, private AI chat... by samspenc on Wednesday August 16, 2023 @06:24PM (#37154690)

Ah fascinating, just curious, what's the technical blocker? I thought most of the Llama models were optimized to run on GPUs?

[ Parent | on HN ]

Re:Show HN: LlamaGPT – Self-hosted, offline, private AI chat... by mayankchhabra on Wednesday August 16, 2023 @07:07PM (#37155070)

It's fairly straightforward to add GPU support when running on the host, but LlamaGPT runs inside a Docker container, and that's where it gets a bit challenging.

[ Parent | on HN ]

Re:Show HN: LlamaGPT – Self-hosted, offline, private AI chat... by stavros on Wednesday August 16, 2023 @08:10PM (#37155556)

It shouldn't, nVidia provides a CUDA Docker plugin that lets you expose your GPU to the container, and it works quite well.

[ Parent | on HN ]

Re:Show HN: LlamaGPT – Self-hosted, offline, private AI chat... by dicriseg on Thursday August 17, 2023 @03:26PM (#37166707)

See above if you're interested in that. It does work quite well, even with nested virtualization (WSL2).

[ Parent | on HN ]

Re:Show HN: LlamaGPT – Self-hosted, offline, private AI chat... by stavros on Thursday August 17, 2023 @03:40PM (#37166884)

I am, thanks!

[ Parent | on HN ]