News for hackers. Stuff that matters.
Re:Ask HN: Affordable hardware for running local large langu... by wokwokwok on Sunday May 05, 2024 @08:08AM (#40264236)
You can use a raspberry pi with 8 GB of ram to run a quantised 7B model (eg. https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGUF), or any cheap stick pc.

For larger models, or GPU accelerated inference, there is no “cheap” solution.

Why do you think everyone is so in love with the 7B models?

It’s not because they’re good. They’re just ok, and it’s expensive to run larger models.

There are no comments on this comment yet.