News for hackers. Stuff that matters.

A comment on How to Run Llama 3 405B on Home Devices? Build AI Cluster. Parent comment

Re:How to Run Llama 3 405B on Home Devices? Build AI Cluster by RealStickman_ on Monday July 29, 2024 @03:41AM (#41098586)
If you just want to play with this a bit, around 250$/month should give you enough metal when renting from cheap VDS/dedicated server providers.
[ Parent | on HN ]
Re:How to Run Llama 3 405B on Home Devices? Build AI Cluster by malux85 on Monday July 29, 2024 @04:48AM (#41098915)
Surely not with GPUs? Could you direct me to them please?
[ Parent | on HN ]
Re:How to Run Llama 3 405B on Home Devices? Build AI Cluster by RealStickman_ on Monday July 29, 2024 @12:19PM (#41101567)
This project only supports CPU inference at the moment, so no, no GPUs. I just estimated what some 256GB or RAM might cost monthly on the low end.
[ Parent | on HN ]
Re:How to Run Llama 3 405B on Home Devices? Build AI Cluster by wkat4242 on Monday July 29, 2024 @06:59AM (#41099453)
250$ is a lot of money to me though. I spent €290 on my 16GB GPU for my AI server but that was once off and I really had to think about it.

I'd love to see a llama model that fits now economically inside 16GB. The 8b is a bit too small when quantised even to 8 bits. A 16-20b model would be perfect.

But I think for 400b models to be viable, the hardware pricing really needs to catch up.

[ Parent | on HN ]
Re:How to Run Llama 3 405B on Home Devices? Build AI Cluster by RealStickman_ on Monday July 29, 2024 @12:21PM (#41101585)
> 250$ is a lot of money to me though. I spent €290 on my 16GB GPU for my AI server but that was once off and I really had to think about it.

Agreed, it's a lot of money and definitely more than I'd be willing to spend. However, compared to running such 400b models on a GPU cluster it's extremely cheap (and much slower)

[ Parent | on HN ]