Reproduce GPT-2 (124M) in llm.c in 90 minutes for $20

Re:Reproduce GPT-2 (124M) in llm.c in 90 minutes for $20 by karpathy on Tuesday May 28, 2024 @02:09PM (#40503665)

Zero To Hero doesn't make it all the way to a chatbot, it stops at pretraining, and even that at a fairly small scale or character-level transformer on TinyShakespeare. I think it's a good conceptual intro but you don't get too too far as a competent chatbot. I think I should be able to improve on this soon.

[ Parent | on HN ]

Re:Reproduce GPT-2 (124M) in llm.c in 90 minutes for $20 by 363849473754 on Tuesday May 28, 2024 @03:04PM (#40504285)

Thanks! So, you are considering expanding the Zero to Hero series to include building a basic GPT-2 toy chatbot? I believe you mentioned in one of the early lectures that you planned to include building a toy version of Dalle. Do you still have plans for that as well?

[ Parent | on HN ]

Re:Reproduce GPT-2 (124M) in llm.c in 90 minutes for $20 by maskil on Tuesday May 28, 2024 @03:39PM (#40504716)

Please do! It's a fantastic series!

[ Parent | on HN ]