Re:Llama.cpp 30B runs with only 6GB of RAM now
by dplavery92 on Friday March 31, 2023 @06:09PM
(#35394356)
Sure, but when one 12gb GPU costs ~$800 new (e.g. for the 3080 LHR), "a couple of dozens" of them is a big barrier to entry to the hobbyist, student, or freelancer. And cloud computing offers an alternative route, but, as stated, distribution introduces a new engineering task, and the month-to-month bills for the compute nodes you are using can still add up surprisingly quickly.
Re:Llama.cpp 30B runs with only 6GB of RAM now
by terafo on Friday March 31, 2023 @06:23PM
(#35394546)
We are talking groups, not individuals. I think it is quite possible for couple of hundreds of people to cooperate and train something at least as big as LLaMa 7B in a week or two.
Re:Llama.cpp 30B runs with only 6GB of RAM now
by webnrrd2k on Friday March 31, 2023 @06:27PM
(#35394585)
Maybe a good candidate for the SETI@home treatment?
Re:Llama.cpp 30B runs with only 6GB of RAM now
by terafo on Friday March 31, 2023 @06:31PM
(#35394635)
It is a good candidate. Tech is good 6-18 months away, though.
Re:Llama.cpp 30B runs with only 6GB of RAM now
by nullsense on Friday March 31, 2023 @07:35PM
(#35395229)
How much faster can we develop the tech if we leverage GPT-4 to do it?
One thing I don't understand: If it's possible to chunk and parallelize it, is it not relatively straightforward to do these chunks sequentially on a single GPU with a roughly linear increase in runtime? Or are the parallelized computations actually interdependent and involving message-passing, making this unfeasible?
Re:Llama.cpp 30B runs with only 6GB of RAM now
by latency-guy2 on Friday March 31, 2023 @10:57PM
(#35396618)
Data moving back and forth from CPU to to bus to GPU and back again for however many chunked model parts you have would increase training time far beyond what you would be willing to invest, not to mention how inefficient and power intensive it is, far more power needed than doing just CPU only or GPU only training. Back to the time part - it's not linear at all. IMO its easily quadratic.
It's not unfeasible, in fact that's how things were done before lots of improvements to the various libraries in essence, many corps still have poorly built pipelines that spend a lot of time in CPU land and not enough in GPU land.
Just an FYI as well - intermediate outputs of models are used in quite a bit of ML, you may see them in some form being used for hyperparameter optimization and searching.
You can't if you have one 12gb gpu. You can if you have couple of dozens. And then petals-style training is possible. It is all very very new and there are many unsolved hurdles, but I think it can be done.