Re:Memorizing Transformers
by visarga on Friday May 20, 2022 @12:49PM
(#31449895)
It's a sparse attention scheme. They store and reuse activations thus "memorising" the past without the need for training. In order to keep the sequence short enough to fit into memory they only recall the k most similar memories from a much larger log.
[ Parent
| on HN ]
There are no comments on this comment yet.