News for hackers. Stuff that matters.

A comment on Ask HN: Which LLMs can run locally on most consumer computers. Parent comment

Re:Ask HN: Which LLMs can run locally on most consumer compu... by FezzikTheGiant on Tuesday May 21, 2024 @05:08AM (#40425883)
Interesting - is it viable do you think to package a llm like that with an existing game and run it locally - I assume it will be intensive to run but wouldn't that eliminate inference costs?
[ Parent | on HN ]
Re:Ask HN: Which LLMs can run locally on most consumer compu... by jaggs on Tuesday May 21, 2024 @06:02AM (#40426353)
Why would you get costs with a local model?
[ Parent | on HN ]
Re:Ask HN: Which LLMs can run locally on most consumer compu... by FezzikTheGiant on Tuesday May 21, 2024 @06:12AM (#40426418)
yeah that's what I'm saying - it would eliminate inference costs. What I was asking is how feasible is it to package these local llms with another standalone app. For ex. a game
[ Parent | on HN ]
Re:Ask HN: Which LLMs can run locally on most consumer compu... by jaggs on Tuesday May 21, 2024 @06:27AM (#40426522)
Oh sorry. Hm..I actually have no idea. It sounds like a neat idea though. :)
[ Parent | on HN ]
Re:Ask HN: Which LLMs can run locally on most consumer compu... by Werewolf255 on Wednesday May 22, 2024 @01:01PM (#40443301)
It would be intensive but it's very doable. You could use koboldcpp or something like that with an exposed endpoint just on the local machine and use that. You'll likely run into issues with GPU vendors and ensuring that you've got the right software versions running, but with some checking, it should be viable. Maybe include a fallback in case the system can't produce results in a timely manner.
[ Parent | on HN ]