News for hackers. Stuff that matters.
Re:Run LLMs at home, BitTorrent‑style by brucethemoose2 on Sunday September 17, 2023 @04:33PM (#37549025)
> and fine‑tune them for your tasks

This is the part that raised my eyebrows.

Finetuning 70B is not just hard, its literally impossible without renting a very expensive cloud instance or buying a PC the price of a house, no matter how long you are willing to wait. I would absolutely contribute to a "llama training horde"

Re:Run LLMs at home, BitTorrent‑style by AaronFriel on Sunday September 17, 2023 @04:39PM (#37549076)
That's true for conventional fine-tuning, but is it the case for parameter efficient fine tuning and qLORA? My understanding is that for a N billion parameter model, fine tuning can occur with a slightly-less-than-N gigabyte of VRAM GPU.

For that 70B parameter model: an A100?

Re:Run LLMs at home, BitTorrent‑style by zacmps on Sunday September 17, 2023 @06:33PM (#37549967)
I think you'd need 2 80GB A100's for unquantised.
Re:Run LLMs at home, BitTorrent‑style by brucethemoose2 on Sunday September 17, 2023 @06:59PM (#37550185)
2x 40/48GB GPUs would be the cheapest. But that's still a very expensive system, especially if you don't have a beefy workstation with 2x PCIe slots just lying around.
Re:Run LLMs at home, BitTorrent‑style by LoganDark on Monday September 18, 2023 @12:32AM (#37552262)
Even mATX boards tend to come with two (full-length) PCIe slots, and that's easy sub-$1k territory. Not exactly a beefy workstation.

Source: have a $200 board in my computer right now with two full-length PCIe slots.

Re:Run LLMs at home, BitTorrent‑style by 7speter on Monday September 18, 2023 @01:40AM (#37552613)
Whats more difficult is trying to cool gpus with 24-48gb of RAM… they all seem to be passively cooled
Re:Run LLMs at home, BitTorrent‑style by LoganDark on Monday September 18, 2023 @01:51AM (#37552679)
Good point, I think most of them are designed for a high-airflow server chassis, with airflow in a direction that a desktop case wouldn't necessarily facilitate (parallel to the card).
Re:Run LLMs at home, BitTorrent‑style by segfaultbuserr on Monday September 18, 2023 @02:17AM (#37552855)
Waterblocks exist for some compute-only GPUs, including the Nvidia A100. Also, there are a few small vendors in China that offer mounting kits that allow you to mod these compute-only GPUs to use off-the-shelf AIO watercoolers. Certainly, not many people are going to take the risk to modify the expensive Nvidia A100, but these solutions are moderately popular among the DIY home lab developers to convert older server cards for home workstation use. Decommissioned Nvidia Tesla P100 or V100 can be purchased cheaply for several hundreds dollars.
Re:Run LLMs at home, BitTorrent‑style by LoganDark on Monday September 18, 2023 @02:50AM (#37553070)
> Decommissioned Nvidia Tesla P100 or V100 can be purchased cheaply for several hundreds dollars.

Meh. If you want 16GB of VRAM for several hundred dollars, can't you just pull a brand new 30-series off the shelf and have ten times more computing power than those old pascal cards? You'll even have more VRAM if you go for the 3080 or 3090. Admittedly, the 3090 is closer to $700 or so, but it should still make a P100 very sad in comparison.

Re:Run LLMs at home, BitTorrent‑style by segfaultbuserr on Monday September 18, 2023 @03:13AM (#37553236)
Yeah, these GPUs became less appealing after the prices of 30-series GPUs have dropped. The price of SXM cards are still somewhat unbeatable though if you have a compatible server motherboard [1]. Nvidia P100s are being sold for as low as $100 each, there are similar savings for the Nvidia V100s. But yeah, a saving around $100 to $200 is not really worthwhile...

Another curious contender is the decommissioned Nvidia CMP series GPUs from miners. For example, the Nvidia CMP 170HX basically uses the same Nvidia A100 PCB with its features downsized or disabled (8 GB VRAM, halved shaders, etc). But interestingly, it seems to preserve the full 1500 GB/s memory bandwidth, making it potentially an interesting card for running memory-bound simulations.

[1] Prices are so low exactly because most people don't. SXM-to-PCIe adapters also exist which cost $100-$200 which is nearly as much as you have saved. It should be trivial to reverse-engineer and make an free and open source version...

Re:Run LLMs at home, BitTorrent‑style by LoganDark on Monday September 18, 2023 @03:19AM (#37553281)
Is it possible to take something like a CMP 170HX and do board-level work to add more memory chips? Or are they not connected to silicon?
Re:Run LLMs at home, BitTorrent‑style by segfaultbuserr on Monday September 18, 2023 @04:03AM (#37553530)
I don't believe it's possible. The HBM2e chips are integrated onto the package of the GPU die, making them impossible to remove or modify in a non-destructive manner.
Re:Run LLMs at home, BitTorrent‑style by brucethemoose2 on Monday September 18, 2023 @10:56PM (#37565137)
I didn't know the CMP had full bandwidth. that would be a an excellent card for smallish networks (like stable diffusion, GANs, audio networks)

...But it doesn't seem to be cheap. Not really worth it over a 4090 for the same price.

Re:Run LLMs at home, BitTorrent‑style by segfaultbuserr on Tuesday September 19, 2023 @03:04AM (#37566535)
It seems that the CMP 170HX is being sold for $500 +/- $100 on the flea markets in China as closed mining farms are dumping any remaining inventory. Not sure if the prices are real, I'm currently trying to purchase some.
Re:Run LLMs at home, BitTorrent‑style by segfaultbuserr on Wednesday September 20, 2023 @06:14AM (#37582445)
I can confirm that the price is real ;-)
Re:Run LLMs at home, BitTorrent‑style by brucethemoose2 on Monday September 18, 2023 @10:54PM (#37565125)
The Quadros/Firepros have blower coolers.
Re:Run LLMs at home, BitTorrent‑style by brucethemoose2 on Monday September 18, 2023 @07:40PM (#37563735)
Not with full x16/x16, though I suppose you don't necessarily need that.
Re:Run LLMs at home, BitTorrent‑style by LoganDark on Tuesday September 19, 2023 @03:57AM (#37566821)
Of course, usually the other PCIe slots are something stupid, but there's still a second full-length one, so this could potentially fit two GPUs with the right power supply.
Re:Run LLMs at home, BitTorrent‑style by pama on Monday September 18, 2023 @09:14AM (#37555650)
If one is training the full 70B parameters, then the total memory usage far exceeds the memory for simply storing the 70B parameters (think derivatives and optimizer parameters such as momentum.) This is the main reason why models are split or why techniques like the fully distributed data sharing are used during training. During training of a distributed model, at every step of the optimizer these multiple-of-70B parameters need to go through a network wire (though not to all nodes, thankfully). As you suggested, LoRA could work well in a distributed setting because the trainable parameters are very small in number (tens of thousand of times less trainable parameters) and the info required to go through the network for non trainable parameters is also small. However, training this model on a single A100 is impractical as it would require mimicking a distributed training buffering things on a TB-sized CPU RAM (or slower) to swap pieces in and out of the model during every step in an otherwise distributed operation (and is not natively supported in existing frameworks to the best of my knowledge, even though one could technically write this code without too much difficulty.)
Re:Run LLMs at home, BitTorrent‑style by Zetobal on Sunday September 17, 2023 @04:35PM (#37549046)
An H100 is maybe a car but not nearly close to a house...
Re:Run LLMs at home, BitTorrent‑style by KomoD on Sunday September 17, 2023 @04:58PM (#37549240)
Maybe not in your area, but it's very doable in other places, like where I live.
Re:Run LLMs at home, BitTorrent‑style by ShamelessC on Sunday September 17, 2023 @08:54PM (#37550964)
You expect me to believe there are other places than where I live?!
Re:Run LLMs at home, BitTorrent‑style by nextaccountic on Sunday September 17, 2023 @05:23PM (#37549481)
Is a single H100 enough?
Re:Run LLMs at home, BitTorrent‑style by brucethemoose2 on Sunday September 17, 2023 @11:17PM (#37551824)
80GB is enough, yeah.

I'm not sure what exact LORA/quantization settings would be ideal, but check out https://github.com/OpenAccess-AI-Collective/axolotl#config

Re:Run LLMs at home, BitTorrent‑style by ioedward on Sunday September 17, 2023 @05:57PM (#37549726)
8 H100s would have enough VRAM to finetune a 70B model.
Re:Run LLMs at home, BitTorrent‑style by akomtu on Sunday September 17, 2023 @05:02PM (#37549275)
What prevents parallel LLM training? If you read book 1 first and then book 2, the resulting update in your knowledge will be the same if you read the books in the reverse order. It seems reasonable to assume that LLM is trained on each book independently, the two deltas in the LLM weights can be just added up.
Re:Run LLMs at home, BitTorrent‑style by whimsicalism on Sunday September 17, 2023 @05:37PM (#37549599)
By the “delta in the LLM weights”, I am assuming you mean the gradients. You are effectively describing large batch training (data parallelism) which is part of the way you can scale up but there are quickly diminishing returns to large batch sizes.
Re:Run LLMs at home, BitTorrent‑style by contravariant on Sunday September 17, 2023 @05:08PM (#37549328)
In ordinary gradient descent the order does matter, since the position changes in between. I think stochastic gradient descent does sum a couple of gradients together sometimes, but I'm not sure what the trade-offs are and if LLMs do so as well.
Re:Run LLMs at home, BitTorrent‑style by ctoth on Sunday September 17, 2023 @06:17PM (#37549849)
This is not at all intuitive to me. It doesn't make sense in a human perspective, as each book changes you. Consider the trivial case of a series, where nothing will make sense if you haven't read the prior books (not that I think they feed it the book corpus in order maybe they should!), but even in a more philosophical sort of way, each book changes you. and the person who reads Harry Potter first and The Iliad second will have a different experience of each. Then, with large language models, we have the concept of grokking something. If grokking happens in the middle of book 1, it is a different model which is reading book 2 and of course the inverse applies.
Re:Run LLMs at home, BitTorrent‑style by eachro on Sunday September 17, 2023 @06:19PM (#37549864)
I'm not sure this is true. For instance, consider reading textbooks for linear algebra and functional analysis out of order. You might still grok the functional analysis if you read it first but you'd be better served by reading the linear algebra one first.
Re:Run LLMs at home, BitTorrent‑style by necroforest on Sunday September 17, 2023 @07:59PM (#37550640)
LLMs are trained in parallel. The model weights and optimizer state are split over a number (possibly thousands) of accelerators.

The main bottleneck to doing distributed training like this is the communication between nodes.

Re:Run LLMs at home, BitTorrent‑style by Grimblewald on Sunday September 17, 2023 @09:00PM (#37551005)
This isn't true. Set up even a simple ANN dense feed forward three layers you know the one. Then keep everything the same for two models you train with the exception of data order. You'll end up with two different models even though you started with the same weights, etc.
Re:Run LLMs at home, BitTorrent‑style by dTal on Monday September 18, 2023 @12:38AM (#37552297)
The "deltas" are calculated by the error in how well the current state of the network predicts the output, backpropagated. Sequential runs are not commutative because the state changes.

Consider the trivial example of training a network to distinguish between sample A and sample B. Give it a hundred As in a row and it just learns "everything is A". Give it a hundred Bs in a row and it relearns "no, everything is B". To train it to distinguish, you must alternate As and Bs (and not too regularly, either!)

Re:Run LLMs at home, BitTorrent‑style by malwrar on Sunday September 17, 2023 @06:07PM (#37549793)
Impossible? It’s just a bunch of math, you don’t need to keep the entire network in memory the whole time.
Re:Run LLMs at home, BitTorrent‑style by brucethemoose2 on Sunday September 17, 2023 @07:02PM (#37550211)
Well, any scheme where weights are dynamically loaded/unloaded from memory enough to fit on a 48GB GPU are so slow that training is basically impractical. Your 70B model would be obsolete by the time the finetuning is done.

Some inference frameworks came up with schemes for just this, and it was horrifically slow.

Re:Run LLMs at home, BitTorrent‑style by pavelstoev on Sunday September 17, 2023 @07:39PM (#37550472)
You can finetune 40B falcon on 4 x A10 with compiler optimization technology from CentML. No changes to the model.
Re:Run LLMs at home, BitTorrent‑style by YetAnotherNick on Monday September 18, 2023 @06:21AM (#37554330)
Finetuning in a distributed way with questionable network would be lot more energy/cost inefficient than doing it with a single node or a well connected cluster. Also, you can finetune 70b model for million tokens for $2 in lambda cloud or <$10 in replicate.