Re:Falcon 40B LLM (which beats Llama) now Apache 2.0
by Timshel on Thursday June 01, 2023 @04:23AM
(#36148709)
Probably this : https://huggingface.co/docs/accelerate/usage_guides/big_mode...
Re:Falcon 40B LLM (which beats Llama) now Apache 2.0
by MacsHeadroom on Friday June 02, 2023 @06:46AM
(#36163480)
Without NVLink different layers are loaded onto individual cards and only one card often has to wait for the other card, slowing down generation. It's still faster than CPU offloading. You can even mix and match GPUs like a 4090 + 3060.
I thought that 4090s were "nerfed" and nvlink support removed - https://www.windowscentral.com/hardware/computers-desktops/n...