Generative Modeling with Sparse Transformers

Re:Generative Modeling with Sparse Transformers by yorwba on Tuesday April 23, 2019 @04:58PM (#19732793)

Using two attention layers with √N inputs to cover a context of size N = √N × √N is somewhat intuitively understandable for image data, since the decomposition corresponds to rows and columns.

But it's quite surprising that this also works for text data, especially that the fixed pattern performs better than the strided one, despite there not being anything analogous to image boundaries in the data.

It'd also be interesting to see what happens for other decompositions, such as 3 layers of ∛N or a logarithmic stack of dilated convolutions.

[ Parent | on HN ]

Re:Generative Modeling with Sparse Transformers by AdamDKing on Tuesday April 23, 2019 @05:19PM (#19732993)

It seems that using "fixed attention" for text would encourage the network to periodically summarize the context so far and put it in that fixed column for the rows below to access.

Maybe the reason "strided attention" didn't work as well is that it would require the network to put this context summary in every column lest the rows below be unable to access it. That would waste features since the summary wouldn't vary much over time but would still be stored in full at each step.

If this is true, the approach they used for images might actually be inefficient in a similar way.

[ Parent | on HN ]