News for hackers. Stuff that matters.
Re:Understanding and Coding the Self-Attention Mechanism of ... by inciampati on Saturday February 11, 2023 @08:45AM (#34752251)
Practical attention implementations don't work over arbitrary length sequences. The universal approximation theorem holds IMO. Information will mix as you go through fully connected MLP layers. Attention is apparently a prior structure that is needed to really reduce training costs.

There are no comments on this comment yet.