Re:I Run LLMs Locally
by fragmede on Sunday December 29, 2024 @04:27PM
(#42543346)
if you're doing serious work, performance is more important than getting a good price/perf ratio, and a pair of 3090s is gonna be faster. It depends on your budget,
however as that configuration is a bit more expensive, however.
[ Parent
| on HN ]
Re:I Run LLMs Locally
by adam_arthur on Sunday December 29, 2024 @04:37PM
(#42543479)
Whether performance or cost is more important depends on your use case. Some tasks that an LLM can do very well may not need to be done often, or even particularly quickly (as in my case).
e.g. LLM as one step of an ETL-style pipeline
Latency of the response really only matters if that response is user facing and is being actively awaited by the user