Re:Graph Neural Networks Use Graphs When They Shouldn't
by eachro on Tuesday September 19, 2023 @12:49PM
(#37572676)
If they don't actually help then the attention weight for that connection would tend to 0, right? Then it becomes a problem of overfitting which we have a large arsenal to combat.
Re:Graph Neural Networks Use Graphs When They Shouldn't
by SpaceManNabs on Tuesday September 19, 2023 @12:50PM
(#37572700)
> If they don't actually help then the attention weight for that connection would tend to 0, right?
Not necessarily. That is what this paper is about. From what I understand, they also consider graph attention networks.
If the problem is that the graph is not sparse enough / not a graph at all, adding more connections doesn't help.
edit: Why doesn't it help? They address this in the paper. There is computational infeasiability problem. The transformer decoder can't be a regular graph if you include masking.