News for hackers. Stuff that matters.
Re:I Run LLMs Locally by ashleyn on Sunday December 29, 2024 @05:07PM (#42543796)
anyone got a guide on setting up and running the business-class stuff (70B models over multiple A100, etc)? i'd be willing to spend the money but only if i could get a good guide on how to set everything up, what hardware goes with what motherboard/ram/cpu, etc.
Re:I Run LLMs Locally by nickpsecurity on Sunday December 29, 2024 @06:22PM (#42544505)
People shared them regularly on r/LocalLlama:

https://www.reddit.com/r/LocalLLaMA/

Search for terms like hardware build, running large models, multiple GPU’s, etc. Many people there have multiple, consumer GPU’s. There’s probably something about running multiple A100’s.

HuggingFace might have tutorials, too.

Warning: If it’s A100’s, most people say to just rent them from the cloud as needed cuz they’re highly costly upfront. If they’re normally idle, then it’s not as cost effective to own them. Some were using services like vast.ai to get rentals cheaper.

Re:I Run LLMs Locally by bubaumba on Sunday December 29, 2024 @09:00PM (#42545662)
it may be even cheaper to use big models through API. Claude, GPT, whatever. Rental is efficient only for big butches, while API is priced per call/size and is cheap for small models.
Re:I Run LLMs Locally by foundry27 on Sunday December 29, 2024 @05:17PM (#42543890)
Aye, there’s the kicker. The correct configuration of hardware resources to run and multiplex large models is just as much of a trade secret as model weights themselves when it comes to non-hobbyist usage, and I wouldn’t be surprised if optimal setups are in many ways deliberately obfuscated or hidden to keep a competitive advantage

Edit: outside the HPC community specifically, I mean

Re:I Run LLMs Locally by codybontecou on Sunday December 29, 2024 @11:13PM (#42546413)
The economic barrier to entry probably has a lot to do with it. I'd happily dig into this problem and share my findings but it's simply too expensive for a hobbyist that isn't specialized in it.
Re:I Run LLMs Locally by talldayo on Sunday December 29, 2024 @05:52PM (#42544229)
I don't think you're going to find much of a guide out there because there isn't really a need for one. You just need a Linux client with the Nvidia drivers installed and some form of CUDA runtime present. You could make that happen in a mini PC, a jailbroken Nintendo Switch, a gaming laptop or a 3600W 4U rackmount. The "happy path" is complicated because you truly have so many functional options.

You don't want an A100 unless you've already got datacenter provisioning at your house and an empty 1U rack. I genuinely cannot stress this enough - these are datacenter cards for a reason. The best bang-for-your buck will be consumer-grade cards like the 3060 and 3090, as well as the bigger devkits like the Jetson Orin.

Re:I Run LLMs Locally by Aurornis on Sunday December 29, 2024 @11:38PM (#42546526)
> anyone got a guide on setting up and running the business-class stuff (70B models over multiple A100, etc)?

If you need a guide, the A100 class cards are certainly not what you want. Running data center level clusters in the home is no walk in the park. You wouldn’t even be able to power a decent sized cluster without getting an electrician involved for proper power. Networking is expensive. Even second-hand it all adds up fast. Don’t underestimate how much it would cost.

Getting some 3090s in a case with a big gaming PSU is the way to go for home use. If you really want A100 level, rent it from a cloud provider. Trying to build your own cluster would be like lighting money on fire for a rapidly depreciating asset.

Re:I Run LLMs Locally by jwrallie on Monday December 30, 2024 @12:20AM (#42546698)
Question: what can you do with the A100s that you cannot to with multiple 3090s?

Or rephrasing for somebody that never tried to experiment with models: which one do you need to have something like ChatGPT-4o (text only) at home?