Re:Llama.cpp 30B runs with only 6GB of RAM now
by amrb on Wednesday April 05, 2023 @11:44AM
(#35456064)
For the life of me I could never fix torch.load,
they'll say just quantization (convert) a model to 4/8bit to make it smaller but you'll get crashes when out of system memory plus no docs..
then you admit defeat by using more swapfile :S
[ Parent
| on HN ]
There are no comments on this comment yet.