News for hackers. Stuff that matters.

A comment on Show HN: AI companions stack – create and host your own AI companions.

Re:Show HN: AI companions stack – create and host your own A... by mcbuilder on Wednesday July 12, 2023 @05:57PM (#36702235)
I hosted my own AI using SillyTavern + text-generation backend (e.g. oogabooga / KoboldAI), just need the power needed to crank the LLM.
[ Parent | on HN ]
Re:Show HN: AI companions stack – create and host your own A... by brucethemoose2 on Wednesday July 12, 2023 @07:27PM (#36703071)
Koboldcpp (with ngrok if you need it) is another excellent self hosting solution.

13b will work on 16GB RAM, and 33b on 32GB RAM, with pretty much any dGPU for a little acceleration and RAM offloading.

Doubly so if you host it as an AI Horde node (so you have priority access to many models through the web browser).

[ Parent | on HN ]