News for hackers. Stuff that matters.

A comment on Reproduce GPT-2 (124M) in llm.c in 90 minutes for $20. Parent comment

Re:Reproduce GPT-2 (124M) in llm.c in 90 minutes for $20 by soraki_soladead on Wednesday May 29, 2024 @12:15AM (#40508400)
Fwiw, that's SwiGLU in #3 above. Swi = Swish = silu. GLU is gated linear unit; the gate construction you describe.
[ Parent | on HN ]

There are no comments on this comment yet.