News for hackers. Stuff that matters.

A comment on Reproduce GPT-2 (124M) in llm.c in 90 minutes for $20. Parent comment

Re:Reproduce GPT-2 (124M) in llm.c in 90 minutes for $20 by simonw on Tuesday May 28, 2024 @03:57PM (#40504909)
> Keep in mind that here we trained for 10B tokens, while GPT-3 models were all trained for 300B tokens. [...] GPT-3 actually didn't change too much at all about the model (context size 1024 -> 2048, I think that's it?).

Andrej, based on that do you have a rough cost estimate for what it would take to train a GPT-3 Ada (350M)? Do you plan to get there with llm.c ?

[ Parent | on HN ]
Re:Reproduce GPT-2 (124M) in llm.c in 90 minutes for $20 by karpathy on Tuesday May 28, 2024 @04:02PM (#40504950)
The 350M model I trained last night was 30B tokens, 14 hours, ~$200. Conveniently, 300B is exactly 10X the tokens so ~$2K would be the estimate. You'd have to wait 140 hours on one box though. Getting an H100 box instead of A100 will already cut the time latency down probably by a factor of 2-3X, for free, even without going to fp8 (which we do plan to support).

So TLDR at this model scale, llm.c is already there functionally, I think, it's a matter of the compute resources and patience. I currently have this one box from Lambda and I have to look around for a few more boxes and merge the pending PR for multi-node training support. Getting all of this into a nice, stable state is probably a good chunk of the pending work right now.

[ Parent | on HN ]