Copilot is based on OpenAI Codex, which is based on GPT-3, which is a transformer model.
Although technically, transformers are mostly GNNs that are "fully connected" (in the graph theory sense), I don't think that supports your speculation here about how GNNs could be used for code analysis since the "tokens" that GPT-3 is trained on are not programming-language syntactic constructs, but sub-word units obtained from natural language (something like WordPiece).
I will say though, I am equally excited by the exact prospect you raised of using something like GNNs for code analysis.
My hunch is that if somebody can figure out a way to make training hierarchical/graph based neural networks very fast, we'll observe the same gains that we did with transformers. But hierarchical/graph based models don't lend themselves to efficient computation.
Definitely check them out! There are also tools that were made available by some of the authors: https://github.com/google-research/python-graphs
Unfortunately, GNNs are lagging LLMs in the code domain. Maybe because
a. LLMs and transformers rulezz OR b. there is far more source code than there are compiled code graphs
Definitely not an expert, just excited to have more tools on which to work with graph data!
[0] Graph Embeddings - Neo4J. Talk by Alicia Frame. https://www.youtube.com/watch?v=oQPCxwmBiWo
In that way, GNNs solve graph problems, the same way CNNs solve image processing problems. Training the GNN is more of an optimisation problem.
Edit: maybe graph theory can help with training on very large graphs but I don't know enough about that.
The transformer comment is interesting. They're very close, and in general, the tricks people use elsewhere are getting translated to GNNs: convolution, attention, ... . But scaling is still happening, so recent couple of years have gotten folks doing 1M-1B level, but not yet LLM scale yet. Critically, the scaling work is relatively recent -- good GPU impl, good samplers, etc -- and with a good trajectory.
We tracked GNNs for years but stayed away until heterogeneity + scaling started to get realistic for commercial workloads, and that's finally happening. Major credit to folks like deepmind, michael bronstein, early graphsage / jure, various individual researchers, and now aws+nvidia engineers for practical engineering evolutions here.
There are two confusing parts for me
1 - The words network and graph are nearly synonymous in this context, and IIRC most neural network architectures are actually graphs that fit some specific pattern. I don't know what makes a 'graph neural network' special (my guess is it has to do with how the layers relate but i don't know)
2 - I almost always see a mention of a graph-related use cases in the context of GNNs. I don't know if there is a fundamental reason for that or if it just so happens that people who have huge graphs worth applying ML to are actually just have really good intuition about how graphs can be leveraged and go that route.
Either that or networks that can solve specific problems on graph structures by more or less general and hence reusable methods. Which could go ways towards the former point, I guess, but that's also dealt eith elsewhere. Just give g-scholar a search and see
I think this already exists in software engineering research, but iirc they were comparing against some code snippets gathered from other open source project or use language models instead of GNN.
Theoretically, it should be possible. There might not exist demos of these sorts of networks yet.
* Predicting the color of a node in a graph, could be used for example speculative devirtualization.
* Predicting edges weight could give us a better estimate of hot basic blocks statically.
* Running performance experiments is as easy as running the benchmark and introducing some metric of performance which you can give back to the GNN to learn from.
Imagine also for debugging and IDEs. I haven't played with copilot, but I imagine that something like guessing the graph based on node labels and on some edges is feasible using GNN? This means that the IDE could try to match the name of your variables, the control flow, and the name of the function to other known functions and could potentially point out differences. Potentially giving us a way to fix logic errors, or better algos. E.g., "Mmm... it looks like you are implementing your own bubble sort from scratch. Would you like to click here and use <insert better sort>."
I am not an expert on GNN, but if anyone has resources for someone to learn more about the state of the art of GNNs (a link to a literature review or something similar) do let me know.