News for hackers. Stuff that matters.
Re:Show HN: less than 650 LOC trainable GPT only using NumPy... by gfaure on Friday November 17, 2023 @02:26PM (#38308734)
Nice! The README mentions `LayerNorm` is implemented here, but while it's in the equivalence tests with PyTorch, I don't see it in the implementation.
Re:Show HN: less than 650 LOC trainable GPT only using NumPy... by dauertewigkeit on Friday November 17, 2023 @03:36PM (#38309767)
It's part of the TensorLi definition where all the magic happens.
Re:Show HN: less than 650 LOC trainable GPT only using NumPy... by joennlae on Friday November 17, 2023 @06:29PM (#38312194)
That is true. I went for a simple implementation of the layer norm and included it in the tensorli definition. But it would have been better to define it as a moduli for clarity.