News for hackers. Stuff that matters.

A comment on Were RNNs All We Needed?. Parent comment

Re:Were RNNs All We Needed? by mickg10 on Friday October 04, 2024 @05:09PM (#41745609)
I.e. ReLU is _piecewise_ linear. That discontinuity that separates the 2 pieces is precisely what makes it non linear. Which is what enables the actual universal approximation.
[ Parent | on HN ]
Re:Were RNNs All We Needed? by quantadev on Friday October 04, 2024 @06:02PM (#41745940)
Which is what I said two replies ago.

Followed by "in some sense it's [ReLU] still even MORE linear than tanh or sigmoid functions are". There's no way you misunderstood that sentence, or took it as my "definition" of linearity...so I guess you just wanted to reaffirm I was correct, again, so thanks.

[ Parent | on HN ]