Re:Reproduce GPT-2 (124M) in llm.c in 90 minutes for $20
by auspiv on Tuesday May 28, 2024 @02:34PM
(#40503954)
Considering it takes 8x A100 GPUs (80GB VRAM) to train GPT-2, I think it'll take far more than a single 4090.
[ Parent
| on HN ]
Re:Reproduce GPT-2 (124M) in llm.c in 90 minutes for $20
by bufo on Tuesday May 28, 2024 @02:48PM
(#40504083)
The RTX 4090 has about the same BF16 Tensor Core TOPs than the A100, assuming 50% MFU (like the A100 40 GB PCIe) it would take 8x longer on 1 RTX 4090 vs 8x A100 80GB SXM, so 12 hours.
Datasheet here for the TOPs https://images.nvidia.com/aem-dam/Solutions/geforce/ada/nvid...
50% MFU should be achievable on the 4090.
Re:Reproduce GPT-2 (124M) in llm.c in 90 minutes for $20
by anthonix1 on Wednesday May 29, 2024 @10:12AM
(#40512217)
Nah, I reproduced on 4x 7900 XTX machine in 8.75 hours, so a single 7900 XTX (costs less than $1k) could do it in under 24 hours. Was hitting 55.4% MFU.
Re:Reproduce GPT-2 (124M) in llm.c in 90 minutes for $20
by benterix on Tuesday June 04, 2024 @04:43AM
(#40572108)
Do you have a link to a description of your setup? I assume there are some quirks on the way and getting this done requires some substantial work. A submission on that would be very interesting, at least to me.