News for hackers. Stuff that matters.
Re:Falcon 40B LLM (which beats Llama) now Apache 2.0 by coolspot on Friday June 02, 2023 @04:01AM (#36162415)
That’s what llama.cpp does, including offload to a disk. It allows you to run models as big as any combination of your VRAM, RAM or disk. But in the end, if it doesn’t fit into GPU VRAM, it will be slow.

For example Guanaco-33B generates ~10 token per second running fully from VRAM of my 3090, the ~1 token/second running from DDR4 RAM of my Ryzen. I would imagine it would do like a token per minute from NVM SSD.

Re:Falcon 40B LLM (which beats Llama) now Apache 2.0 by anonymousDan on Friday June 02, 2023 @06:51AM (#36163514)
Is it memory/disk bandwidth bound or latency bound?
Re:Falcon 40B LLM (which beats Llama) now Apache 2.0 by mike_hearn on Friday June 02, 2023 @08:41AM (#36164304)
Memory bandwidth bound.