chaitjo

Author here. I recently defended my PhD thesis on Geometric Deep Learning. This post summarizes 4 years of research into three main questions covering:

- Theoretical expressivity of 3D molecular representations (Geometric WL test).

- Unified generative modelling for crystals and molecules (Diffusion Transformers).

- Inverse design of RNA structure/function with wet-lab validation.

Happy to answer any questions about the research or the PhD process!

AI for science is very hot right now. I wrote some personal reflections about being physically embedded in a world-leading molecular biology lab as an AI researcher: learning to communicate with experimentalists, back-breaking wet lab work, scientific rigour, and skin in the game.

his is a lightweight, hackable way to run AlphaFold 3 that lets you experiment without the massive MSA databases or Docker overhead. Perfect for tinkering on your laptop or single GPU server - just install, drop in your sequences, and start predicting. No 100GB downloads, no container bloat, just pure structure prediction you can play with in a Jupyter notebook.

We’ve just released a “Hitchhiker’s Guide” for getting started with deep neural nets for 3D structural biology & chemistry -- we think it will be useful for newcomers to start their learning journey on the core architectures powering recent breakthroughs of graph ML in protein design, material discovery, molecular simulations, and more!

PDF: https://arxiv.org/abs/2312.07511

Authors: Alexandre Duval, Simon V. Mathis, Chaitanya K. Joshi, Victor Schmidt, Santiago Miret, Fragkiskos Malliaros, Taco Cohen, Pietro Liò, Yoshua Bengio, and Michael Bronstein

Geometric GNNs are an emerging class of GNNs for spatially embedded graphs in scientific and engineering applications, s.a. biomolecular structure, material science, and physical simulations. Notable examples include SchNet, DimeNet, Tensor Field Networks, and E(n) Equivariant GNNs.

https://github.com/chaitjo/geometric-gnn-dojo/blob/main/geom...

This notebook and repository aims to serve as a 'Geometric GNNs 101' introduction for newcomers.

We walk through the basics of GNNs, Geometric Deep Learning, and the PyTorch Geometric library for implementing these concepts.

Our goal is to help students understand how theory/equations connect to real code.

Indeed, the two papers came out within months of each other iirc. The GAT paper discusses Transformers in the context of stabilizing the learning of attention mechanisms.

Of course, this connection may be trivial to most people, but I hadn't seen a post on this before. So I decided to write one for myself as I studied these architectures.

In graph terms, Positional encodings are useful for adding sequential/temporal properties to each node in the graph. Indeed, there are works on position-aware GNNs.

Not necessarily. In fact, applying the Transformer/GNN on a full graph is seen by some people as 'discovering' or identifying some useful underlying graph structure. We had an interesting discussion on this on Twitter: https://twitter.com/PetarV_93/status/1233820829852618754

Thanks for highlighting! (Admin of the lab website)

I am guessing he means that Transformers 'discover' parse trees in sentences, as they operate on fully-connected graphs but end up focusing on the most important connections through attention.