Re:I Run LLMs Locally
by bloomingkales on Sunday December 29, 2024 @03:46PM
(#42542921)
Honestly I think if you just want to do inferencing the 7600xt and rx6800 have 16gb at $300 and $400 on Amazon. It's gonna be my stop gap until whatever. The RX6800 has better memory bandwidth than the 4060ti (think it matches the 4070).
[ Parent
| on HN ]
AMD GPUs are a fantastic deal until you hit a problem. Some models/frameworks it works great. Others, not so much.
Re:I Run LLMs Locally
by bloomingkales on Sunday December 29, 2024 @04:00PM
(#42543057)
For sure but I think people on the fine tuning/training/stable diffusion side are more concerned with that. They make a big fuss about this and basically talk people out of a perfectly good and well priced 16gb vram card that literally works out of the box with ollama, lmstudio for text inferencing.
Kind of one of the reasons AMD is a sleeper stock for me. If people only knew.