News for hackers. Stuff that matters.
Re:RedPajama (Open LLaMA Replication) 3B and 7B models released by int_19h on Saturday May 06, 2023 @04:49AM (#35839462)
It's not even that bad. Core i7-12700K with DDR5 gives me ~1 word per second on llama-30b - that is fast enough for real-time chat, with some patience. And things are even better on M1/M2 Macs.
Re:RedPajama (Open LLaMA Replication) 3B and 7B models released by Joeri on Saturday May 06, 2023 @05:47AM (#35839753)
The critical factor seems to be the ability to fit the whole model in RAM (--mlock option in oobabooga). With Apple's RAM prices most M1/M2 owners probably don't have the 32 GB RAM required to fit a 4bit 30B model.
Re:RedPajama (Open LLaMA Replication) 3B and 7B models released by Semaphor on Saturday May 06, 2023 @08:59AM (#35840924)
I have 64 GB RAM, but only a Ryzen 5 3600, and the larger models are very slow ;)