Re:Generative Modeling with Sparse Transformers
by AdamDKing on Tuesday April 23, 2019 @06:36PM
(#19733685)
You seem to be saying this work is based on convolutional neural networks. That's incorrect. It uses the same attention mechanisms from natural language processing which involve no convolution operations.
Convolutions have a different set of weights for each position offset (with a fixed window size), and reuse those weights across the entire input space.
Transformer-based networks like this work compute attention functions between the current position's encoding and every previous position, then use the outputs to compute a weighted sum of the encodings at those positions. Hence they can look at an arbitrarily large window and the number of parameters they have is independent of the size of that window.
Re:Generative Modeling with Sparse Transformers
by skdotdan on Tuesday April 23, 2019 @06:14PM
(#19733508)
Are Transformers based on convolutions?
Re:Generative Modeling with Sparse Transformers
by joe_the_user on Tuesday April 23, 2019 @06:32PM
(#19733644)
"Convolutional neural networks", whose connection to other means of convolution is a bit tenuous.
The innovation here is that the transformer is compressed, allowing the system to deal with longer sequences.