News for hackers. Stuff that matters.
Re:A comprehensive guide to running Llama 2 locally by zakki on Tuesday July 25, 2023 @06:30PM (#36870580)
Is there any specific settings to make 2x3090 work together?
Re:A comprehensive guide to running Llama 2 locally by kwerk on Tuesday July 25, 2023 @11:59PM (#36872913)
Interested in this too
Re:A comprehensive guide to running Llama 2 locally by eurekin on Wednesday July 26, 2023 @03:55AM (#36874407)
Not really? I just got those cards in separate PCI slots and the Exllama_hf handles spreading the load internally. No NVLink bridge in particular. I use the "20,22" memory split so that the display card has some room for the framebuffer to handle display
Re:A comprehensive guide to running Llama 2 locally by davidy123 on Wednesday July 26, 2023 @03:12PM (#36883290)
Do you mean you don't use NVLink or just use one that works? I am under the impression it is being phased out ("PCIe 5 is fast enough") and some kits don't use it.
Re:A comprehensive guide to running Llama 2 locally by eurekin on Thursday July 27, 2023 @03:39AM (#36890221)
I don't use NVLink