News for hackers. Stuff that matters.

A comment on Differential Transformer. Parent comment

Re:Differential Transformer by smallnamespace on Tuesday October 08, 2024 @06:40PM (#41782523)
> using max(0, exp(x)-1) instead of exp(x)

Won't this cause the gradient to vanish on the left half, causing problems with training?

[ Parent | on HN ]
Re:Differential Transformer by espadrine on Wednesday October 09, 2024 @02:41AM (#41785150)
That is a concern that is shared with ReLU. But since the weights are shared across the context/minibatch, perhaps that would not be an issue, similar to ReLU.
[ Parent | on HN ]