Re:Ask HN: Recommendations for Local LLMs in 2024: Private a...
by sosuke on Sunday April 07, 2024 @11:35AM
(#39961454)
Shooting from the hip here I’d say $800 would be a good start for all new parts. You might get down to $500 with used parts.
The most expensive piece in either build is a video card. You want to be able to load the LLM file into your video card RAM.
I just got a 16gb new card for this. I can load up to 34b models on it but poorly. Anything 13b or less runs perfectly. A 12gb card would be able to run 7b models and with the right training I think 7b models can be awesome.
The most expensive piece in either build is a video card. You want to be able to load the LLM file into your video card RAM.
I just got a 16gb new card for this. I can load up to 34b models on it but poorly. Anything 13b or less runs perfectly. A 12gb card would be able to run 7b models and with the right training I think 7b models can be awesome.
New 4060ti Used 3090