Re:Understanding and Coding the Self-Attention Mechanism of ...
by inciampati on Saturday February 11, 2023 @08:45AM
(#34752251)
Practical attention implementations don't work over arbitrary length sequences. The universal approximation theorem holds IMO. Information will mix as you go through fully connected MLP layers. Attention is apparently a prior structure that is needed to really reduce training costs.
[ Parent
| on HN ]
There are no comments on this comment yet.