News for hackers. Stuff that matters.
Re:Llama from Scratch (or how to implement a paper without c... by dannymi on Wednesday August 09, 2023 @12:25PM (#37065028)
>These are placed on the outputs of a neuron and are one of those weird mathematical hacks where I can't really explain why it's needed,

Because when you compose linear functions you get linear functions. So having linear everything is a waste of all layers but one.

In order for this not to happen, you need nonlinearity.

Re:Llama from Scratch (or how to implement a paper without c... by mike_hearn on Thursday August 10, 2023 @03:35AM (#37073129)
Thanks!