Re:Reproduce GPT-2 (124M) in llm.c in 90 minutes for $20
by anthonix1 on Tuesday May 28, 2024 @08:10PM
(#40506887)
It converges similarly on smaller datasets.
About to kick off a training from scratch run on the same fineweb-10B, which at 324k toks/sec should take about 8.6 hours. And with my kWh cost, that is about $2.50 cost to train.
Will report back tomorrow when the training has finished..
About to kick off a training from scratch run on the same fineweb-10B, which at 324k toks/sec should take about 8.6 hours. And with my kWh cost, that is about $2.50 cost to train.
Will report back tomorrow when the training has finished..