News for hackers. Stuff that matters.

A comment on Building a fully local LLM voice assistant to control my smart home. Parent comment

Re:Building a fully local LLM voice assistant to control my ... by behnamoh on Saturday January 13, 2024 @06:29PM (#38985798)
you can spawn multiple llama.cpp servers and query them simultaneously. It’s actually better this way because you get to run different models for different purposes or do sanity checks via a second model.
[ Parent | on HN ]
Re:Building a fully local LLM voice assistant to control my ... by JohnTheNerd on Saturday January 13, 2024 @06:34PM (#38985835)
that is correct, however I am already using all of my VRAM. it would mean I have to degrade my model quality. I instead decided that I would rather have one solid model, and have all my use cases tied to that one model. using RAM instead proved to be problematic for the reasons I mentioned above.

if I had any free VRAM at all, I would fit faster-whisper before I touch any other LLM lol

[ Parent | on HN ]