News for hackers. Stuff that matters.
Re:Memorizing Transformers by lucidrains on Friday May 20, 2022 @11:36AM (#31448899)
have an implementation of this over at https://github.com/lucidrains/memorizing-transformers-pytorc..., for any researcher exploring retrieval and memory with attention networks
Re:Memorizing Transformers by knrz on Friday May 20, 2022 @11:49AM (#31449070)
Dude your repo’s are great, marvellous code quality too for cutting edge papers. Keep it up!
Re:Memorizing Transformers by lucidrains on Friday May 20, 2022 @12:20PM (#31449503)
hey thanks! :^) hope someone makes the next big discovery with them
Re:Memorizing Transformers by silencedogood3 on Friday May 20, 2022 @12:34PM (#31449673)
Neat! Can you explain what the KNn is doing? I can’t quite follow the paper.
Re:Memorizing Transformers by visarga on Friday May 20, 2022 @12:49PM (#31449895)
It's a sparse attention scheme. They store and reuse activations thus "memorising" the past without the need for training. In order to keep the sequence short enough to fit into memory they only recall the k most similar memories from a much larger log.