News for hackers. Stuff that matters.
Re:A comprehensive guide to running Llama 2 locally by eurekin on Tuesday July 25, 2023 @05:37PM (#36869929)
Yes, it runs totally fine. I ran it in Oobabooga/text generation web ui.

Model: TheBloke_Llama-2-70B-chat-GPTQ from https://huggingface.co/TheBloke/Llama-2-70B-chat-GPTQ

ExLlama_HF loader gpu split 20,22, context size 2048

Re:A comprehensive guide to running Llama 2 locally by zakki on Tuesday July 25, 2023 @06:30PM (#36870580)
Is there any specific settings to make 2x3090 work together?
Re:A comprehensive guide to running Llama 2 locally by kwerk on Tuesday July 25, 2023 @11:59PM (#36872913)
Interested in this too
Re:A comprehensive guide to running Llama 2 locally by eurekin on Wednesday July 26, 2023 @03:55AM (#36874407)
Not really? I just got those cards in separate PCI slots and the Exllama_hf handles spreading the load internally. No NVLink bridge in particular. I use the "20,22" memory split so that the display card has some room for the framebuffer to handle display
Re:A comprehensive guide to running Llama 2 locally by davidy123 on Wednesday July 26, 2023 @03:12PM (#36883290)
Do you mean you don't use NVLink or just use one that works? I am under the impression it is being phased out ("PCIe 5 is fast enough") and some kits don't use it.
Re:A comprehensive guide to running Llama 2 locally by eurekin on Thursday July 27, 2023 @03:39AM (#36890221)
I don't use NVLink