News for hackers. Stuff that matters.
Re:Llama.cpp 30B runs with only 6GB of RAM now by amrb on Wednesday April 05, 2023 @11:44AM (#35456064)
For the life of me I could never fix torch.load, they'll say just quantization (convert) a model to 4/8bit to make it smaller but you'll get crashes when out of system memory plus no docs.. then you admit defeat by using more swapfile :S

There are no comments on this comment yet.