News for hackers. Stuff that matters.
Re:I Run LLMs Locally by Aurornis on Sunday December 29, 2024 @11:38PM (#42546526)
> anyone got a guide on setting up and running the business-class stuff (70B models over multiple A100, etc)?

If you need a guide, the A100 class cards are certainly not what you want. Running data center level clusters in the home is no walk in the park. You wouldn’t even be able to power a decent sized cluster without getting an electrician involved for proper power. Networking is expensive. Even second-hand it all adds up fast. Don’t underestimate how much it would cost.

Getting some 3090s in a case with a big gaming PSU is the way to go for home use. If you really want A100 level, rent it from a cloud provider. Trying to build your own cluster would be like lighting money on fire for a rapidly depreciating asset.

Re:I Run LLMs Locally by jwrallie on Monday December 30, 2024 @12:20AM (#42546698)
Question: what can you do with the A100s that you cannot to with multiple 3090s?

Or rephrasing for somebody that never tried to experiment with models: which one do you need to have something like ChatGPT-4o (text only) at home?