News for hackers. Stuff that matters.
Re:I Run LLMs Locally by elorant on Sunday December 29, 2024 @04:38PM (#42543502)
I consider the RTX 4060 Ti as the best entry level GPU for running small models. It has 16GBs of RAM which gives you plenty of space for running large context windows and Tensor Cores which are crucial for inference. For larger models probably multiple RTX 3090s since you can buy them on the cheap on the second hand market.

I don’t have experience with AMD cards so I can’t vouch for them.

Re:I Run LLMs Locally by fnqi8ckfek on Sunday December 29, 2024 @05:21PM (#42543925)
I know nothing about gpus. Should I be assuming that when people say "ram" in the context of gpus they always mean vram?
Re:I Run LLMs Locally by elorant on Sunday December 29, 2024 @06:55PM (#42544770)
Not always, because system RAM also has to be equally adequate, but mostly yes, it's about the total VRAM of the GPU(s).
Re:I Run LLMs Locally by layer8 on Sunday December 29, 2024 @05:27PM (#42543972)
“GPU with xx RAM” means VRAM, yes.