Full forward pass of GPT-2 in one file of pure CUDA

Posted by tosh on Wednesday April 10, 2024 @02:53PM
from the github.com dept.

https://github.com/karpathy/llm.c/blob/master/train_gpt2.cu

( 4 comments | 54 points | on HN )

Re:Full forward pass of GPT-2 in one file of pure CUDA by yinser on Wednesday April 10, 2024 @03:34PM (#39994874)

I feel guilty hoping such a great researcher continues making educational content but damn it Andrej puts out incredible material.

[ Parent | on HN ]

Re:Full forward pass of GPT-2 in one file of pure CUDA by latchkey on Wednesday April 10, 2024 @03:33PM (#39994858)

I would love for someone to run HIPIFY[0] on this and prove that it works (or doesn't). I'd do it myself, but I am unable to at this time.

[0] https://github.com/ROCm-Developer-Tools/HIPIFY

[ Parent | on HN ]

Re:Full forward pass of GPT-2 in one file of pure CUDA by noxs on Wednesday April 10, 2024 @05:24PM (#39995882)

this is quite the difference I saw among different ML Engineers during my work.. most can only do python (more or less like a frameworker) and some amazing ones can go all the way down to the CUDA to squeeze out the performance.

[ Parent | on HN ]

Re:Full forward pass of GPT-2 in one file of pure CUDA by skilled on Saturday April 20, 2024 @03:00AM (#40095431)

An update from Karpathy:

> Our single file of 2,000 ~clean lines of C/CUDA code now trains GPT-2 (124M) on GPU at speeds ~matching PyTorch (fp32, no flash attention)

> On my A100 I'm seeing 78ms/iter for llm.c and 80ms/iter for PyTorch. Keeping in mind this is fp32, with no flash attention yet, and slightly stale PyTorch (2.1.0).

https://twitter.com/karpathy/status/1781387674978533427

[ Parent | on HN ]