Starting from scratch: Training a 30M Topological Transformer

Re:Starting from scratch: Training a 30M Topological Transfo... by andai on Sunday January 18, 2026 @12:23PM (#46669825)

This is interesting. Has there been more research into this architecture? I hear about it once every few years but it always seems like a niche / experimental thing. But based on the graph in their blog post you'd expect every company to be using this.

[ Parent | on HN ]

Re:Starting from scratch: Training a 30M Topological Transfo... by tuned on Monday January 19, 2026 @01:18AM (#46675574)

This is a novel re-interpretation of the Transformer, based on my previous research made with a library called `arrowspace`.

It is somehow what is called a "Grassmann-like flow" but without the Plucker embedding, or also similar to what is done in DavisTensor but relying on spectral Laplacian instead of purely geometric distances.

The problem with a lot of stuff done before is that it focuses on dense representations. This architecture is focuses on sparse representation and provides a new approximation computation based on energy-informed graphs.

[ Parent | on HN ]