Reproduce GPT-2 (124M) in llm.c in 90 minutes for $20

Re:Reproduce GPT-2 (124M) in llm.c in 90 minutes for $20 by karpathy on Tuesday May 28, 2024 @01:41PM (#40503324)

My understanding and suspicion is mostly less than you think. Llama 3 architecture has the following changes on GPT-2:

1. delete the absolute positional encoding and replace with RoPE

2. delete all biases in all layers (in LayerNorms, they turn into RMSNorm)

3. GeLU -> SwiGLU non-linearity in the MLP

4. longer context length

5. architecture hyperparameter changes, e.g. slightly different aspect ratios

And there was a paper that I can't find the reference to anymore that claimed that if you train long enough, the gap becomes even lower. Possibly because the absolutely positional encoding has enough time to train more fully, where as the RoPE layer benefits from the "inductive bias" it adds in the earlier stages of training.

But I don't have full confidence on the above claim, maybe someone has tried or has better/concrete reference.

[ Parent | on HN ]

Re:Reproduce GPT-2 (124M) in llm.c in 90 minutes for $20 by jorlow on Tuesday May 28, 2024 @03:50PM (#40504843)

Note llama's feed forward is a bit different too:

  self.w2(F.silu(self.w1(x)) * self.w3(x))

I.e. the nonlinearity is a gate.

https://github.com/meta-llama/llama3/blob/14aab0428d3ec3a959...

[ Parent | on HN ]

Re:Reproduce GPT-2 (124M) in llm.c in 90 minutes for $20 by soraki_soladead on Wednesday May 29, 2024 @12:15AM (#40508400)

Fwiw, that's SwiGLU in #3 above. Swi = Swish = silu. GLU is gated linear unit; the gate construction you describe.

[ Parent | on HN ]