News for hackers. Stuff that matters.

A comment on Transformers Without Normalization. Parent comment

Re:Transformers Without Normalization by gricardo99 on Saturday March 15, 2025 @01:48AM (#43370299)
from the abstract

  By incorporating DyT, Transformers without normalization can match or exceed the performance of their normalized counterparts, mostly without hyperparameter tuning.
[ Parent | on HN ]
Re:Transformers Without Normalization by gdiamos on Sunday March 16, 2025 @08:01AM (#43378332)
Sure, but why would one prefer tanh instead of normalization layers if they have the same accuracy?
[ Parent | on HN ]