Re:Nvidia's 900 tons of GPU muscle bulks up server market, s...
by alecco on Tuesday September 19, 2023 @01:16PM
(#37573092)
GPU cores are powerful but you need to feed them data at very high speeds and large models require to pass around a lot of data. This is why Nvidia datacenter products have NVLink and Infiniband so Big Corp can run their massive models.
In a recent paper I was reading they were bragging they "only" needed 8 x A100 to train their model.
Renting a 4090 24GB costs around $.40/hr. Renting A100 80GB costs around $3.4/hr. For 3x memory and 2x bandwidth.
To fit a 13B model in a 4090 you need to quantize it. Imagine a 130B or 200B model. And never mind training, which needs a lot more memory.
The divide is crazy. If you are not FAANG you are "GPU-poor".
In a recent paper I was reading they were bragging they "only" needed 8 x A100 to train their model.
Renting a 4090 24GB costs around $.40/hr. Renting A100 80GB costs around $3.4/hr. For 3x memory and 2x bandwidth.
To fit a 13B model in a 4090 you need to quantize it. Imagine a 130B or 200B model. And never mind training, which needs a lot more memory.
The divide is crazy. If you are not FAANG you are "GPU-poor".