Re:Transformers Are Graph Neural Networks
by YetAnotherNick on Saturday September 12, 2020 @07:42PM
(#24456896)
Actually we do have few mechanisms for long term memory like Neural Turing Machine, which has explicit memory cells which neural network could read and write. I think the only thing that is holding back NTM is that it is computationally not efficient like fixed sized context transformer.
[ Parent
| on HN ]
Re:Transformers Are Graph Neural Networks
by Muller20 on Sunday September 13, 2020 @03:50AM
(#24459207)
What's holding back NTM is that they are hard to train, even worse than RNNs. They are not much less efficient than a Transformer. Instead, Transformer has all the advantages of the NTM but it is much easier to train.
Actually, the way I see it, Transformer is a direct descendent of memory-based architectures (NTM, MemNet, stack-based RNNs...) that is both expressive and easy to train.