News for hackers. Stuff that matters.
Re:Falcon 40B LLM (which beats Llama) now Apache 2.0 by dheera on Friday June 02, 2023 @02:31AM (#36161781)
Stupid question but for feed-forward models why do we not yet have some kind of CPU RAM memory swap mechanism? Why is Pytorch still trying to load the whole damn model into GPU RAM at once and then complaining when it can't, instead of swapping portions of the model to CPU RAM, or hell, even SSD?

Sure, it might be a lot slower, but that's a lot better than "I give up, go buy $20K worth of hardware"

Re:Falcon 40B LLM (which beats Llama) now Apache 2.0 by coolspot on Friday June 02, 2023 @04:01AM (#36162415)
That’s what llama.cpp does, including offload to a disk. It allows you to run models as big as any combination of your VRAM, RAM or disk. But in the end, if it doesn’t fit into GPU VRAM, it will be slow.

For example Guanaco-33B generates ~10 token per second running fully from VRAM of my 3090, the ~1 token/second running from DDR4 RAM of my Ryzen. I would imagine it would do like a token per minute from NVM SSD.

Re:Falcon 40B LLM (which beats Llama) now Apache 2.0 by anonymousDan on Friday June 02, 2023 @06:51AM (#36163514)
Is it memory/disk bandwidth bound or latency bound?
Re:Falcon 40B LLM (which beats Llama) now Apache 2.0 by mike_hearn on Friday June 02, 2023 @08:41AM (#36164304)
Memory bandwidth bound.
Re:Falcon 40B LLM (which beats Llama) now Apache 2.0 by sebzim4500 on Friday June 02, 2023 @09:44AM (#36164952)
I haven't run the numbers, but I would expect that doing that would make it slower than just running on the CPU.