News for hackers. Stuff that matters.
Re:Run LLMs at home, BitTorrent‑style by Zetobal on Sunday September 17, 2023 @04:35PM (#37549046)
An H100 is maybe a car but not nearly close to a house...
Re:Run LLMs at home, BitTorrent‑style by KomoD on Sunday September 17, 2023 @04:58PM (#37549240)
Maybe not in your area, but it's very doable in other places, like where I live.
Re:Run LLMs at home, BitTorrent‑style by ShamelessC on Sunday September 17, 2023 @08:54PM (#37550964)
You expect me to believe there are other places than where I live?!
Re:Run LLMs at home, BitTorrent‑style by nextaccountic on Sunday September 17, 2023 @05:23PM (#37549481)
Is a single H100 enough?
Re:Run LLMs at home, BitTorrent‑style by brucethemoose2 on Sunday September 17, 2023 @11:17PM (#37551824)
80GB is enough, yeah.

I'm not sure what exact LORA/quantization settings would be ideal, but check out https://github.com/OpenAccess-AI-Collective/axolotl#config

Re:Run LLMs at home, BitTorrent‑style by ioedward on Sunday September 17, 2023 @05:57PM (#37549726)
8 H100s would have enough VRAM to finetune a 70B model.