News for hackers. Stuff that matters.
Re:Falcon 40B LLM (which beats Llama) now Apache 2.0 by sireat on Thursday June 01, 2023 @02:41AM (#36148196)
How is it possible to run this model on 2x 4090s ?

I thought that 4090s were "nerfed" and nvlink support removed - https://www.windowscentral.com/hardware/computers-desktops/n...

Re:Falcon 40B LLM (which beats Llama) now Apache 2.0 by Timshel on Thursday June 01, 2023 @04:23AM (#36148709)
Re:Falcon 40B LLM (which beats Llama) now Apache 2.0 by MacsHeadroom on Friday June 02, 2023 @06:46AM (#36163480)
Without NVLink different layers are loaded onto individual cards and only one card often has to wait for the other card, slowing down generation. It's still faster than CPU offloading. You can even mix and match GPUs like a 4090 + 3060.