News for hackers. Stuff that matters.

A comment on Karpathy: Let's reproduce GPT-2 (1.6B): one 8XH100 node 24h $672 in llm.c.

Re:Karpathy: Let's reproduce GPT-2 (1.6B): one 8XH100 node 2... by alecco on Thursday July 11, 2024 @03:22PM (#40939710)
Also https://x.com/karpathy/status/1811467135279104217#m
[ Parent | on HN ]
Re:Karpathy: Let's reproduce GPT-2 (1.6B): one 8XH100 node 2... by iforiq on Thursday July 11, 2024 @03:30PM (#40939769)
How much did gpt2 training cost when it came out in 2019?
[ Parent | on HN ]
Re:Karpathy: Let's reproduce GPT-2 (1.6B): one 8XH100 node 2... by ozr on Thursday July 11, 2024 @03:36PM (#40939823)
About $50,000.
[ Parent | on HN ]
Re:Karpathy: Let's reproduce GPT-2 (1.6B): one 8XH100 node 2... by withinboredom on Thursday July 11, 2024 @03:37PM (#40939832)
They probably spent more on the training data, to be honest. They had to get it the hard way.
[ Parent | on HN ]