News for hackers. Stuff that matters.

A comment on A Stack of Feed-Forward Layers Does Surprisingly Well on ImageNet. Parent comment

Re:A Stack of Feed-Forward Layers Does Surprisingly Well on ... by BobbyJo on Tuesday January 17, 2023 @08:21PM (#34422161)
Wouldn't attention networks use less computation for the same number of weights? Feed forward networks have higher connectivity no?

If a network is the same size, less computationally demanding, and gives you 3% improvement, it seems extremely worthwhile, especially given the (mostly) diminishing returns of just adding more weights/layers.

[ Parent | on HN ]
Re:A Stack of Feed-Forward Layers Does Surprisingly Well on ... by jeffreyrogers on Tuesday January 17, 2023 @09:08PM (#34422563)
It is less general though, so this is interesting because it shows you can learn a lot without adding very much structure.
[ Parent | on HN ]