Re:Falcon 40B LLM (which beats Llama) now Apache 2.0
by coolspot on Friday June 02, 2023 @04:01AM
(#36162415)
That’s what llama.cpp does, including offload to a disk. It allows you to run models as big as any combination of your VRAM, RAM or disk. But in the end, if it doesn’t fit into GPU VRAM, it will be slow.
For example Guanaco-33B generates ~10 token per second running fully from VRAM of my 3090, the ~1 token/second running from DDR4 RAM of my Ryzen. I would imagine it would do like a token per minute from NVM SSD.
Re:Falcon 40B LLM (which beats Llama) now Apache 2.0
by anonymousDan on Friday June 02, 2023 @06:51AM
(#36163514)
Is it memory/disk bandwidth bound or latency bound?
Re:Falcon 40B LLM (which beats Llama) now Apache 2.0
by mike_hearn on Friday June 02, 2023 @08:41AM
(#36164304)
Memory bandwidth bound.
Re:Falcon 40B LLM (which beats Llama) now Apache 2.0
by sebzim4500 on Friday June 02, 2023 @09:44AM
(#36164952)
I haven't run the numbers, but I would expect that doing that would make it slower than just running on the CPU.
Sure, it might be a lot slower, but that's a lot better than "I give up, go buy $20K worth of hardware"