News for hackers. Stuff that matters.
Re:Hopfield Networks Is All You Need by visarga on Sunday May 02, 2021 @01:53AM (#27012647)
If you read the title, it only refers to the multi-head-attention part of BERT, excluding the feed forward and skip connections, hence calling it "Pure Attention".

> Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth

This does not prove the original title was wrong, and this paper is not a counter, but an analysis of a submodule which helps better understanding transformers.

There are no comments on this comment yet.