News for hackers. Stuff that matters.
Re:A comprehensive guide to running Llama 2 locally by lolinder on Tuesday July 25, 2023 @06:28PM (#36870540)
VRAM is what gets you up to the larger model sizes, and 24GB isn't enough to load the full 70B even at 4 bits, you need at least 35 and some extra for the context. So it depends a lot on what you want to do—fine tuning will take even more as I understand it.

The card's speed will affect your performance, but I don't know enough about different graphics cards to tell you specifics.

Re:A comprehensive guide to running Llama 2 locally by ycombmehair on Tuesday July 25, 2023 @09:01PM (#36871852)
How would an APU, such as 5700g with up to 128gb of system ram perform when allocating it as vram? Is this a cost effective way of using running this on a budget?
Re:A comprehensive guide to running Llama 2 locally by NoMoreNicksLeft on Tuesday July 25, 2023 @09:30PM (#36872065)
Well, 48gb is better than nothing at least. And it has the potential (if we get the build right) to drop a second A6000 card into it with the nvlink module (I think this does allow you to effectively have 96gb) later.