Re:The Coming of Local LLMs
by jrm4 on Tuesday April 11, 2023 @12:33PM
(#35527512)
As I see these things come out, it feels like there's not a lot of discussion on which hardware (that isn't one of the fancy new Macs?) As in, there might be a lot of graphics cards out there that could be used here? Is it only Nvidia still, is AMD a possibility? Maybe I'm missing something on how the tech works?
[ Parent
| on HN ]
This has a list of models and their VRAM requirements
https://www.reddit.com/r/LocalLLaMA/comments/11o6o3f/how_to_...
30B llama needs a 3090 or 4090. 13B I think you can get away with a 3/4080. If you have 64 gigs of ram and a beefy CPU you can run even 65B, but boy it's slow.
13B is pretty meh, but 30B is great, if not quite Chatgpt. But I can ask it why my highschool geometry teacher was such a cunt and it will happily discuss the matter without reservation. Very therapeutic.