if you're doing serious work, performance is more important than getting a good price/perf ratio, and a pair of 3090s is gonna be faster. It depends on your budget,
however as that configuration is a bit more expensive, however.
Re:I Run LLMs Locally
by adam_arthur on Sunday December 29, 2024 @04:37PM
(#42543479)
Whether performance or cost is more important depends on your use case. Some tasks that an LLM can do very well may not need to be done often, or even particularly quickly (as in my case).
e.g. LLM as one step of an ETL-style pipeline
Latency of the response really only matters if that response is user facing and is being actively awaited by the user
You can easily use the MacMini as a hub for running the LLM while you do work on your main computer (and it won't eat up your system resources or turn your primary computer into a heater)
I hope that more non-mac PCs come out optimized for high RAM SoC, I'm personally not a huge Apple fan but use them begrudgingly.
Also your $900 quote is a used/refurbished GPU. I've had plenty of GPUs burn out on me in the old days, not sure how it is nowadays, but that's a lot to pay for a used part IMO