Re:Falcon 40B LLM (which beats Llama) now Apache 2.0
by anonymousDan on Friday June 02, 2023 @06:51AM
(#36163514)
Is it memory/disk bandwidth bound or latency bound?
Re:Falcon 40B LLM (which beats Llama) now Apache 2.0
by mike_hearn on Friday June 02, 2023 @08:41AM
(#36164304)
Memory bandwidth bound.
For example Guanaco-33B generates ~10 token per second running fully from VRAM of my 3090, the ~1 token/second running from DDR4 RAM of my Ryzen. I would imagine it would do like a token per minute from NVM SSD.