News for hackers. Stuff that matters.
Re:Run LLMs at home, BitTorrent‑style by nextaccountic on Sunday September 17, 2023 @05:23PM (#37549481)
Is a single H100 enough?
Re:Run LLMs at home, BitTorrent‑style by brucethemoose2 on Sunday September 17, 2023 @11:17PM (#37551824)
80GB is enough, yeah.

I'm not sure what exact LORA/quantization settings would be ideal, but check out https://github.com/OpenAccess-AI-Collective/axolotl#config