Starting from scratch: Training a 30M Topological Transformer

Re:Starting from scratch: Training a 30M Topological Transfo... by oofbey on Sunday January 18, 2026 @03:11PM (#46671672)

The big bet with this technique is in having a fixed (non learned) matrix which converts the tokens latent space to the linear attention space. So you can kinda cheat and say your model is small because a bunch of the smarts are in this fixed big graph laplacian matrix L.

So how do you scale this up from a toy problem? Well that L would Have to get bigger. And it’s hard to imagine it being useful if L is not trained. Then it starts to look a lot more like a conventional transformer, but probably harder to train, with the benefit of smaller KV caches. (Half the size - not a massive win.)

So overall doesn’t seem to me like it’s gonna amount to anything.

[ Parent | on HN ]

Re:Starting from scratch: Training a 30M Topological Transfo... by tuned on Monday January 19, 2026 @01:30AM (#46675624)

the idea is to have a lot of "narrow" models to work with RAG instead of one model for all the knowledge domains or also distil the metadata that is currently in enterprise Knowledge Graphs

[ Parent | on HN ]

Re:Starting from scratch: Training a 30M Topological Transfo... by tuned on Monday January 19, 2026 @02:05AM (#46675806)

also: precomputing a sparse Laplacian for N vectors at dimension D (NxD) is infinitely cheaper (if using `arrowspace`, my previous paper) than computing distances on the same full dense vectors billions of times. There are published tests that compute a Laplacian on 300Kx384 space in 500 secs on a laptop on CPU. So it is a trade-off: potentially few minutes of pretaining or hours of dot-product on dense matrices

[ Parent | on HN ]