Transformers Are Graph Neural Networks

Re:Transformers Are Graph Neural Networks by justlexi93 on Monday March 09, 2020 @04:06AM (#22523771)

Graph attention networks were inspired by transformers. I am pretty sure they explicitly say it in the paper.

[ Parent | on HN ]

Re:Transformers Are Graph Neural Networks by chaitjo on Tuesday March 17, 2020 @09:00PM (#22612504)

Indeed, the two papers came out within months of each other iirc. The GAT paper discusses Transformers in the context of stabilizing the learning of attention mechanisms.

Of course, this connection may be trivial to most people, but I hadn't seen a post on this before. So I decided to write one for myself as I studied these architectures.

[ Parent | on HN ]

Re:Transformers Are Graph Neural Networks by sqrt17 on Monday March 09, 2020 @05:28AM (#22524095)

Maybe a better way to put it would be "Graph Neural Networks are a Generalization of Transformers". The formulation of "X are Y" indeed suggests that you could just have stuck with Y instead of using the special case X, whereas in reality they took Y and added something to make it more general.

[ Parent | on HN ]

Re:Transformers Are Graph Neural Networks by huffmsa on Monday March 09, 2020 @07:21AM (#22524500)

Yeah but even without attention, they're pretty similar.

Sentences can be reasonable modeled as a mostly 1 directional graph.

[ Parent | on HN ]