Ask HN: Which LLMs can run locally on most consumer computers

Re:Ask HN: Which LLMs can run locally on most consumer compu... by Isuckatcode on Tuesday May 21, 2024 @03:43PM (#40432913)

Here [1] is a reference to the token/sec of Llama 3 on different apple hardware. You can evaluate if this is an acceptable performance for your agents. I would assume the token/sec would be much lower if the LLM agent is running along the side as the game would also be using a portion of the CPU and GPU. I think this is something that you need to test out on your own to determine its usability.

You can also look into lower parameter models (3B for example) to determine if the balance between accuracy and performance fits under your usecase.

>Is there a way to reliably package these models with existing games and make them run locally? This would virtually make inference free right?

I don't have any knowledge on game dev so I can comment on this but yes, packaging it locally would make the inference free.

[1] https://github.com/ggerganov/llama.cpp/discussions/4167

[ Parent | on HN ]

Re:Ask HN: Which LLMs can run locally on most consumer compu... by smarm52 on Tuesday May 21, 2024 @03:49PM (#40432997)

Well since OP doesn't seem to want to: Thank you for your response.

I came across this thread while doing some research, and it's been helpful.

(I hate how common Tragedy of the Commons is. =/)

[ Parent | on HN ]

Re:Ask HN: Which LLMs can run locally on most consumer compu... by FezzikTheGiant on Wednesday May 22, 2024 @12:25AM (#40437326)

What? chill out buddy - there's such a thing as timezones I was just sleeping

[ Parent | on HN ]

Re:Ask HN: Which LLMs can run locally on most consumer compu... by FezzikTheGiant on Wednesday May 22, 2024 @12:26AM (#40437335)

Thanks! This is helpful. I was thinking about the phi models - those might be useful for this task. Will look into how those can be run locally as well

[ Parent | on HN ]

Re:Ask HN: Which LLMs can run locally on most consumer compu... by jlokier on Wednesday May 22, 2024 @02:33PM (#40444447)

I just ran phi3:mini[1] with Ollama on an Apple M3 Max laptop, on battery set to "Low power" (mentioned because that makes some things run more slowly). phi3:mini output roughly 15-25 words/second. The token rate is higher but I don't have an easy way to measure that.

Then llama3:8b[2]. It output 28 words/second. This is higher despite the larger model, perhaps because llama3 obeyed my request to use short words.

Then mixtral:8x7b[3]. That output 10.5 words/second. It looked like 2 tokens/word, as the pattern was quite repetitive and visible, but again I have no easy way to measure it.

That was on battery, set to "Low power" mode, and I was impressed that even with mixtral:8x7b, the fans didn't come on at all for the first 2 minutes of continuous output. Total system power usage peaked at 44W, of which about 38W was attributable to the GPU.

[1] https://ollama.com/library/phi3 [2] https://ollama.com/library/llama3 [3] https://ollama.com/library/mixtral

[ Parent | on HN ]