Re:The dark side of Graph Neural Networks
by legothief on Wednesday June 29, 2022 @02:19PM
(#31923837)
I'm also quite excited about that - there's existing research, quite a few papers that are using graph-based models for MLOnCode:
https://proceedings.neurips.cc/paper/2021/file/c2937f3a1b3a1...
https://arxiv.org/abs/2203.05181
https://arxiv.org/abs/2005.02161
https://arxiv.org/abs/2012.07023
https://arxiv.org/abs/2005.10636v2
https://arxiv.org/abs/2106.10918
Definitely check them out! There are also tools that were made available by some of the authors: https://github.com/google-research/python-graphs
Re:The dark side of Graph Neural Networks
by algo_trader on Wednesday June 29, 2022 @04:03PM
(#31925049)
Are these papers somehow "curated" or "recommended"?!
Unfortunately, GNNs are lagging LLMs in the code domain. Maybe because
a. LLMs and transformers rulezz OR b. there is far more source code than there are compiled code graphs
Copilot is based on OpenAI Codex, which is based on GPT-3, which is a transformer model.
Although technically, transformers are mostly GNNs that are "fully connected" (in the graph theory sense), I don't think that supports your speculation here about how GNNs could be used for code analysis since the "tokens" that GPT-3 is trained on are not programming-language syntactic constructs, but sub-word units obtained from natural language (something like WordPiece).
I will say though, I am equally excited by the exact prospect you raised of using something like GNNs for code analysis.
My hunch is that if somebody can figure out a way to make training hierarchical/graph based neural networks very fast, we'll observe the same gains that we did with transformers. But hierarchical/graph based models don't lend themselves to efficient computation.