Re:How to Run Llama 3 405B on Home Devices? Build AI Cluster
by atemerev on Monday July 29, 2024 @03:23PM
(#41102975)
Well, I live in an apartment in Switzerland, we have different standards.
Re:How to Run Llama 3 405B on Home Devices? Build AI Cluster
by angoragoats on Monday July 29, 2024 @03:42PM
(#41103110)
Yeah, makes sense. I was just trying to point out that "you won't have enough elecricity to run it at home" is not true for many single-family homeowners.
I'm not familiar with the typical Swiss apartment, so I'm sorry if this is a stupid question, but do you not think that your apartment could sustain a ~15A load at 230V? That's how much you'd need for ten RTX 4090s at full power.
Re:How to Run Llama 3 405B on Home Devices? Build AI Cluster
by atemerev on Monday July 29, 2024 @05:02PM
(#41103773)
It is not just the video cards. CPUs, mainboards, cooling systems, losses/inefficiencies… E.g. 4x4090 Comino Grando workstation is rated for 2811W, which is OK. 8x will probably trip my circuit breaker already, unless I plug it into higher-power circuit for kitchen appliances.
Re:How to Run Llama 3 405B on Home Devices? Build AI Cluster
by angoragoats on Monday July 29, 2024 @05:57PM
(#41104179)
Yep, in the US you’d need to use at least two separate circuits as well. I’m just saying that there probably is sufficient electricity for a crazy 10+ GPU setup, in most single family houses at least.
Re:How to Run Llama 3 405B on Home Devices? Build AI Cluster
by BizarroLand on Tuesday July 30, 2024 @03:12PM
(#41113024)
If you're going that far out you might as well just tap into your breaker box and add a 40a 110v circuit. As long as you can run your cluster near the breaker box it would not be a huge expense to do that, and if like most homes your breaker is in the garage then that would make venting the excess heat out fairly simple.
Re:How to Run Llama 3 405B on Home Devices? Build AI Cluster
by angoragoats on Tuesday July 30, 2024 @05:51PM
(#41114490)
I don’t condone doing this under any circumstances without first consulting an electrician. Lots of things could go wrong; for example, 40A @ 120V is not a very common configuration, and most standard household receptacles are rated for only 20A at most. Consult an electrician, folks.
Re:How to Run Llama 3 405B on Home Devices? Build AI Cluster
by BizarroLand on Wednesday July 31, 2024 @01:04PM
(#41120986)
Thanks, I did mean to imply that you would be paying an electrician to do the work to code, but I did also overlook that someone who wants to run a 405b Llama ai at home might be more of a diy'er than the average person.
Re:How to Run Llama 3 405B on Home Devices? Build AI Cluster
by Kirby64 on Tuesday July 30, 2024 @07:39PM
(#41115168)
Or just run your PSU off 240V. Much more standard supply, and most switched mode supplies already support 240V. They’re typically more efficient, too.
Re:How to Run Llama 3 405B on Home Devices? Build AI Cluster
by angoragoats on Wednesday July 31, 2024 @04:06PM
(#41122925)
That works for sure, though it's less common in the US to have a 240v circuit available (with an appropriate receptacle) anywhere you might put a large server. Using two 120v circuits, you're more likely to be able to set it up with what you have now, without requiring any electrical work.
Re:How to Run Llama 3 405B on Home Devices? Build AI Cluster
by Kirby64 on Wednesday July 31, 2024 @04:12PM
(#41122963)
The comparison was a 120V 40A circuit. A 240V-20A socket would be more common than a 120V 50A socket. Both technically exist, but a 240V 20A outlet would be cheaper, safer, and more common. A 240V 20A outlet is actually somewhat common for larger window mounted AC units.
Re:How to Run Llama 3 405B on Home Devices? Build AI Cluster
by angoragoats on Wednesday July 31, 2024 @07:29PM
(#41124638)
> The comparison was a 120V 40A circuit.
Yes, which was a suggestion in reply to my comment mentioning the use of two standard 120v circuits, which I still maintain is the easiest and cheapest path for most people.
The average home (in the United States at least) has more than enough power to run 10 GPUs, or even more, with plenty left over for other appliances. Back of the napkin math is that ten high-end GPUs, at 350W each, need approximately 30-35A of power, and these days the bare minimum service in a home is 100A (and many homes have more than this; mine has 200A, for example). This could be accomplished on two standard 15A circuits, or perhaps more comfortably two 20A circuits.
You can also throttle your cards down to conserve power; I know that 3090s set to max power of 225W only lose ~5% of performance when running LLM workloads, while saving ~125W of power at full load.