News for hackers. Stuff that matters.
Re:Llama.cpp 30B runs with only 6GB of RAM now by terafo on Friday March 31, 2023 @05:35PM (#35393979)
AI training has very high minimum requirements to get in the door. If your GPU has 12GB of VRAM and your model and gradients require 13GB, you can't train the model. CPUs don't have this limitation but they are ridiculously inefficient for any training task. There are techniques like ZeRO to give pagefile-like state partitioning to GPU training, but that requires additional engineering.

You can't if you have one 12gb gpu. You can if you have couple of dozens. And then petals-style training is possible. It is all very very new and there are many unsolved hurdles, but I think it can be done.

Re:Llama.cpp 30B runs with only 6GB of RAM now by dplavery92 on Friday March 31, 2023 @06:09PM (#35394356)
Sure, but when one 12gb GPU costs ~$800 new (e.g. for the 3080 LHR), "a couple of dozens" of them is a big barrier to entry to the hobbyist, student, or freelancer. And cloud computing offers an alternative route, but, as stated, distribution introduces a new engineering task, and the month-to-month bills for the compute nodes you are using can still add up surprisingly quickly.
Re:Llama.cpp 30B runs with only 6GB of RAM now by terafo on Friday March 31, 2023 @06:23PM (#35394546)
We are talking groups, not individuals. I think it is quite possible for couple of hundreds of people to cooperate and train something at least as big as LLaMa 7B in a week or two.
Re:Llama.cpp 30B runs with only 6GB of RAM now by webnrrd2k on Friday March 31, 2023 @06:27PM (#35394585)
Maybe a good candidate for the SETI@home treatment?
Re:Llama.cpp 30B runs with only 6GB of RAM now by terafo on Friday March 31, 2023 @06:31PM (#35394635)
It is a good candidate. Tech is good 6-18 months away, though.
Re:Llama.cpp 30B runs with only 6GB of RAM now by nullsense on Friday March 31, 2023 @07:35PM (#35395229)
How much faster can we develop the tech if we leverage GPT-4 to do it?
Re:Llama.cpp 30B runs with only 6GB of RAM now by 3np on Friday March 31, 2023 @08:47PM (#35395800)
One thing I don't understand: If it's possible to chunk and parallelize it, is it not relatively straightforward to do these chunks sequentially on a single GPU with a roughly linear increase in runtime? Or are the parallelized computations actually interdependent and involving message-passing, making this unfeasible?
Re:Llama.cpp 30B runs with only 6GB of RAM now by latency-guy2 on Friday March 31, 2023 @10:57PM (#35396618)
Data moving back and forth from CPU to to bus to GPU and back again for however many chunked model parts you have would increase training time far beyond what you would be willing to invest, not to mention how inefficient and power intensive it is, far more power needed than doing just CPU only or GPU only training. Back to the time part - it's not linear at all. IMO its easily quadratic.

It's not unfeasible, in fact that's how things were done before lots of improvements to the various libraries in essence, many corps still have poorly built pipelines that spend a lot of time in CPU land and not enough in GPU land.

Just an FYI as well - intermediate outputs of models are used in quite a bit of ML, you may see them in some form being used for hyperparameter optimization and searching.