Ask HN: Which LLMs can run locally on most consumer computers

Re:Ask HN: Which LLMs can run locally on most consumer compu... by FezzikTheGiant on Tuesday May 21, 2024 @02:15PM (#40431842)

Is there a way to reliably package these models with existing games and make them run locally? This would virtually make inference free right?

What I think is, from my limited understanding about this field, if smaller models can run on consumer hardware reliably and speedily that would be a game changer.

[ Parent | on HN ]

Re:Ask HN: Which LLMs can run locally on most consumer compu... by talldayo on Wednesday May 22, 2024 @01:13PM (#40443441)

> This would virtually make inference free right?

Not really. Inference is never "free" unless you cache the result (which is just a static output) or unless you reduce complexity (which yields procedurally less-usable outputs).

[ Parent | on HN ]

Re:Ask HN: Which LLMs can run locally on most consumer compu... by FezzikTheGiant on Wednesday May 22, 2024 @02:04PM (#40444038)

Can you explain further? Why would it not be free if it's running locally

[ Parent | on HN ]

Re:Ask HN: Which LLMs can run locally on most consumer compu... by talldayo on Wednesday May 22, 2024 @02:30PM (#40444379)

I thought you meant "free" in terms of computational cost; running local is technically free of charge, but also requires a lot of processing power. Inferecing a properly-sized LLM will potentially starve the rest of your software from GPU/CPU/memory access, so you have to plan accordingly.

[ Parent | on HN ]