Re:Llama 3.1 Official Launch
by bick_nyers on Wednesday July 24, 2024 @12:01PM
(#41058401)
Agreed, but there are other costs associated with supporting 10-16x GPUs that may not necessarily happen with say 6 GPUs. Having to go from single socket (or Threadripper) to dual socket, PCIE bifurcation, PLX risers, etc.
Not necessarily saying that Quadros are cheaper, just that there's more to the calculation when trying to run 405B size models at home
Re:Llama 3.1 Official Launch
by angoragoats on Wednesday July 24, 2024 @03:41PM
(#41061067)
The system I outlined in my other post [0] has ten GPUs and does not require dual socket CPUs as far as I'm aware. It could likely scale easily to 14 GPUs as well (assuming you have sufficient power), with an x8/x8 bifurcation adapter installed in each PCIe slot. This is pushing the limits of the PCIe subsystem I'm sure, but you could also likely scale up to 28 GPUs, again assuming sufficient power, by simply bifurcating at x4/x4/x4/x4 vs x8/x8.
I think it should work as-is with the components listed, but if you disagree please let me know!
I don't have any actual figures to back this up, but my gut tells me that the fact that enterprise GPUs are an order of magnitude (at least) more expensive than, say a, 3090, means that the payback period of them has got to be pretty long. I also wonder whether setting the max power on a 3090 to a lower than default value (as I suggest in my other post) has a significant effect on the average W/token.