News for hackers. Stuff that matters.
Re:Run LLMs at home, BitTorrent‑style by LoganDark on Monday September 18, 2023 @02:50AM (#37553070)
> Decommissioned Nvidia Tesla P100 or V100 can be purchased cheaply for several hundreds dollars.

Meh. If you want 16GB of VRAM for several hundred dollars, can't you just pull a brand new 30-series off the shelf and have ten times more computing power than those old pascal cards? You'll even have more VRAM if you go for the 3080 or 3090. Admittedly, the 3090 is closer to $700 or so, but it should still make a P100 very sad in comparison.

Re:Run LLMs at home, BitTorrent‑style by segfaultbuserr on Monday September 18, 2023 @03:13AM (#37553236)
Yeah, these GPUs became less appealing after the prices of 30-series GPUs have dropped. The price of SXM cards are still somewhat unbeatable though if you have a compatible server motherboard [1]. Nvidia P100s are being sold for as low as $100 each, there are similar savings for the Nvidia V100s. But yeah, a saving around $100 to $200 is not really worthwhile...

Another curious contender is the decommissioned Nvidia CMP series GPUs from miners. For example, the Nvidia CMP 170HX basically uses the same Nvidia A100 PCB with its features downsized or disabled (8 GB VRAM, halved shaders, etc). But interestingly, it seems to preserve the full 1500 GB/s memory bandwidth, making it potentially an interesting card for running memory-bound simulations.

[1] Prices are so low exactly because most people don't. SXM-to-PCIe adapters also exist which cost $100-$200 which is nearly as much as you have saved. It should be trivial to reverse-engineer and make an free and open source version...

Re:Run LLMs at home, BitTorrent‑style by LoganDark on Monday September 18, 2023 @03:19AM (#37553281)
Is it possible to take something like a CMP 170HX and do board-level work to add more memory chips? Or are they not connected to silicon?
Re:Run LLMs at home, BitTorrent‑style by segfaultbuserr on Monday September 18, 2023 @04:03AM (#37553530)
I don't believe it's possible. The HBM2e chips are integrated onto the package of the GPU die, making them impossible to remove or modify in a non-destructive manner.
Re:Run LLMs at home, BitTorrent‑style by brucethemoose2 on Monday September 18, 2023 @10:56PM (#37565137)
I didn't know the CMP had full bandwidth. that would be a an excellent card for smallish networks (like stable diffusion, GANs, audio networks)

...But it doesn't seem to be cheap. Not really worth it over a 4090 for the same price.

Re:Run LLMs at home, BitTorrent‑style by segfaultbuserr on Tuesday September 19, 2023 @03:04AM (#37566535)
It seems that the CMP 170HX is being sold for $500 +/- $100 on the flea markets in China as closed mining farms are dumping any remaining inventory. Not sure if the prices are real, I'm currently trying to purchase some.
Re:Run LLMs at home, BitTorrent‑style by segfaultbuserr on Wednesday September 20, 2023 @06:14AM (#37582445)
I can confirm that the price is real ;-)