Re:Falcon 40B LLM (which beats Llama) now Apache 2.0
by amstan on Friday June 02, 2023 @01:52AM
(#36161534)
40B is pretty large, right? I expect it would take 70GB or so of RAM to run it. That's some expensive hardware ($10,000 or more).
[ Parent
| on HN ]
Re:Falcon 40B LLM (which beats Llama) now Apache 2.0
by esquire_900 on Friday June 02, 2023 @02:05AM
(#36161633)
Some people use second hand P40 GPUs, which go for around 200-300$. Combine 3 of them with SLI and you've got 72GB of VRAM for less then $1000
[dead]
Re:Falcon 40B LLM (which beats Llama) now Apache 2.0
by esquire_900 on Thursday June 15, 2023 @05:37AM
(#36338207)
http://nonint.com/ has some interesting posts about how he build a custom server to house 8 GPU's (3090's in this case). You're right that that will set you back more than $1000, though I was only referring to the GPU's themselves.
Re:Falcon 40B LLM (which beats Llama) now Apache 2.0
by amstan on Friday June 02, 2023 @03:09AM
(#36162045)
Woah, that's a cool direction. Thank you! I'll explore this.
Re:Falcon 40B LLM (which beats Llama) now Apache 2.0
by washadjeffmad on Friday June 02, 2023 @07:14AM
(#36163658)
P40s are kind of a meme. Using ggmls has roughly the same performance at a fraction of the wattage on a dual-channel DDR5 system.
I still use GPTQ for 30B, but even CPU generates quickly enough at q5_1 on modern hardware.
Re:Falcon 40B LLM (which beats Llama) now Apache 2.0
by esperent on Friday June 02, 2023 @01:57AM
(#36161574)
I briefly looked at prices a few days ago, I think you can rent an 80gb GPU for about $2.50 an hour. Then you just pay while you're using it.
Someone else might be better able to confirm the pricing, but in any case you don't need to purchase the hardware.
Re:Falcon 40B LLM (which beats Llama) now Apache 2.0
by amstan on Friday June 02, 2023 @03:09AM
(#36162042)
I'm less interested in using someone else's computer (not as much, but similar to how i'm disinterested in an API from someone like OpenAI), would rather pay the upfront hardware cost than worry about how many tokens i generate (kind of hinders creativity and excitment about it).
Re:Falcon 40B LLM (which beats Llama) now Apache 2.0
by Kelamir on Friday June 02, 2023 @02:06AM
(#36161639)
> https://github.com/ggerganov/llama.cpp/issues/1602#issuecomm...
Here somebody quantized it down to 29929.56MB .
Re:Falcon 40B LLM (which beats Llama) now Apache 2.0
by valvar on Friday June 02, 2023 @01:56AM
(#36161569)
That much good RAM in itself isn't super expensive. So does the rest of the hardware have to be particularly powerful?
Re:Falcon 40B LLM (which beats Llama) now Apache 2.0
by esperent on Friday June 02, 2023 @01:58AM
(#36161588)
It has to be GPU RAM from my understanding, unless you're happy to wait several minutes/hours for each response.
Re:Falcon 40B LLM (which beats Llama) now Apache 2.0
by GaggiX on Friday June 02, 2023 @02:00AM
(#36161599)
From what I can find online LLAMA-65B 4-bit quantized can run 1 token/s on a Ryzen 7 3700X (using llama.cpp).
Re:Falcon 40B LLM (which beats Llama) now Apache 2.0
by dheera on Friday June 02, 2023 @02:31AM
(#36161781)
Stupid question but for feed-forward models why do we not yet have some kind of CPU RAM memory swap mechanism? Why is Pytorch still trying to load the whole damn model into GPU RAM at once and then complaining when it can't, instead of swapping portions of the model to CPU RAM, or hell, even SSD?
Sure, it might be a lot slower, but that's a lot better than "I give up, go buy $20K worth of hardware"
Re:Falcon 40B LLM (which beats Llama) now Apache 2.0
by coolspot on Friday June 02, 2023 @04:01AM
(#36162415)
That’s what llama.cpp does, including offload to a disk. It allows you to run models as big as any combination of your VRAM, RAM or disk. But in the end, if it doesn’t fit into GPU VRAM, it will be slow.
For example Guanaco-33B generates ~10 token per second running fully from VRAM of my 3090, the ~1 token/second running from DDR4 RAM of my Ryzen. I would imagine it would do like a token per minute from NVM SSD.
Re:Falcon 40B LLM (which beats Llama) now Apache 2.0
by anonymousDan on Friday June 02, 2023 @06:51AM
(#36163514)
Is it memory/disk bandwidth bound or latency bound?
Re:Falcon 40B LLM (which beats Llama) now Apache 2.0
by mike_hearn on Friday June 02, 2023 @08:41AM
(#36164304)
Memory bandwidth bound.
Re:Falcon 40B LLM (which beats Llama) now Apache 2.0
by sebzim4500 on Friday June 02, 2023 @09:44AM
(#36164952)
I haven't run the numbers, but I would expect that doing that would make it slower than just running on the CPU.
Re:Falcon 40B LLM (which beats Llama) now Apache 2.0
by GaggiX on Friday June 02, 2023 @01:56AM
(#36161567)
With 4-bit quantization should take less than 40GB of VRAM/RAM.
Re:Falcon 40B LLM (which beats Llama) now Apache 2.0
by gymbeaux on Friday June 02, 2023 @01:55AM
(#36161562)
70GB of RAM would cost around $150 these days depending on how you get there. 64GB (2x32GB) of DDR4 is around $140 then another 8GB stick would be around $15.
Used DDR3 ECC would be roughly half that.
Re:Falcon 40B LLM (which beats Llama) now Apache 2.0
by Semaphor on Friday June 02, 2023 @01:56AM
(#36161568)
I think they are talking about VRAM
Re:Falcon 40B LLM (which beats Llama) now Apache 2.0
by drdaeman on Friday June 02, 2023 @01:59AM
(#36161593)
Anything that runs on GPU can be run on CPU, the only question is how much slower it's gonna be.
Re:Falcon 40B LLM (which beats Llama) now Apache 2.0
by Semaphor on Friday June 02, 2023 @02:05AM
(#36161625)
I know, I have a weak GPU but 64 GB RAM. Using those models works, but it’s more "ask a question, then do something else for a while, while your fans spin up" ;)
Re:Falcon 40B LLM (which beats Llama) now Apache 2.0
by amstan on Friday June 02, 2023 @03:13AM
(#36162071)
I've been running vicunda-13B on a workstation that can be aquired from ebay for about $500. It's slow compared to online services, but probably slightly faster than text to speech would recite its output, so plenty.
Falcon 40B is probably too much for it, but apparently there's similar cheap hardware that could work.