Search for terms like hardware build, running large models, multiple GPU’s, etc. Many people there have multiple, consumer GPU’s. There’s probably something about running multiple A100’s.
HuggingFace might have tutorials, too.
Warning: If it’s A100’s, most people say to just rent them from the cloud as needed cuz they’re highly costly upfront. If they’re normally idle, then it’s not as cost effective to own them. Some were using services like vast.ai to get rentals cheaper.
Re:I Run LLMs Locally
by bubaumba on Sunday December 29, 2024 @09:00PM
(#42545662)
it may be even cheaper to use big models through API. Claude, GPT, whatever. Rental is efficient only for big butches, while API is priced per call/size and is cheap for small models.
https://www.reddit.com/r/LocalLLaMA/
Search for terms like hardware build, running large models, multiple GPU’s, etc. Many people there have multiple, consumer GPU’s. There’s probably something about running multiple A100’s.
HuggingFace might have tutorials, too.
Warning: If it’s A100’s, most people say to just rent them from the cloud as needed cuz they’re highly costly upfront. If they’re normally idle, then it’s not as cost effective to own them. Some were using services like vast.ai to get rentals cheaper.