News for hackers. Stuff that matters.
Re:Falcon 40B LLM (which beats Llama) now Apache 2.0 by Semaphor on Friday June 02, 2023 @01:56AM (#36161568)
I think they are talking about VRAM
Re:Falcon 40B LLM (which beats Llama) now Apache 2.0 by drdaeman on Friday June 02, 2023 @01:59AM (#36161593)
Anything that runs on GPU can be run on CPU, the only question is how much slower it's gonna be.
Re:Falcon 40B LLM (which beats Llama) now Apache 2.0 by Semaphor on Friday June 02, 2023 @02:05AM (#36161625)
I know, I have a weak GPU but 64 GB RAM. Using those models works, but it’s more "ask a question, then do something else for a while, while your fans spin up" ;)
Re:Falcon 40B LLM (which beats Llama) now Apache 2.0 by amstan on Friday June 02, 2023 @03:13AM (#36162071)
I've been running vicunda-13B on a workstation that can be aquired from ebay for about $500. It's slow compared to online services, but probably slightly faster than text to speech would recite its output, so plenty.

Falcon 40B is probably too much for it, but apparently there's similar cheap hardware that could work.

Re:Falcon 40B LLM (which beats Llama) now Apache 2.0 by gymbeaux on Monday June 05, 2023 @05:38PM (#36204174)
oop you're right... I know once the model is trained they run on the CPU/regular RAM