Re:Android phone hits 24GB of RAM, as much as a 13-inch MacB...
by brucethemoose2 on Thursday July 06, 2023 @03:37PM
(#36621623)
Thats barely enough for LLaMA 33b.
Q3_K_S 33b fits on my 16GB laptop + 6GB GPU (so 22GB minus some overhead from the split + linux), but its extremely tight and only works up to ~800 context.
Q3_K_S 33b fits on my 16GB laptop + 6GB GPU (so 22GB minus some overhead from the split + linux), but its extremely tight and only works up to ~800 context.