Starting from scratch: Training a 30M Topological Transformer

Re:Starting from scratch: Training a 30M Topological Transfo... by Herring on Sunday January 18, 2026 @03:23PM (#46671776)

I'd say try the nanogpt speedrun. It's much easier to train and gives you a better comparison vs optimized systems.

https://github.com/KellerJordan/modded-nanogpt

[ Parent | on HN ]

Re:Starting from scratch: Training a 30M Topological Transfo... by nickpsecurity on Sunday January 18, 2026 @08:27PM (#46674038)

Labs were also competing to train BERT's for $20 or less. People still use them a lot, too.

https://www.databricks.com/blog/mosaicbert

I'll add they should do a number of small, training runs with different architectures and data mixes. That proves generalization.

[ Parent | on HN ]

Re:Starting from scratch: Training a 30M Topological Transfo... by naasking on Sunday January 18, 2026 @09:13PM (#46674334)

The linked paper tested nanoGPT with this new transformer:

https://www.techrxiv.org/users/685780/articles/1375955-topol...

[ Parent | on HN ]

Re:Starting from scratch: Training a 30M Topological Transfo... by tuned on Monday January 19, 2026 @01:14AM (#46675557)

thanks for linking.

Yes the paper compares the new architecture (that is also a fork of my implementation of nanoGPT) with Karpathy's nanoGPT. There are also links to the code and bench used.

[ Parent | on HN ]

Re:Starting from scratch: Training a 30M Topological Transfo... by Herring on Monday January 19, 2026 @01:07PM (#46682400)

Note I didn't say Karpathy's nanoGPT, I said use the speedrun.

Transformers are universal function approximators. When well-tuned, they often start to approximate other innovations. Not always, thank god, but often enough that you have to be careful.

[ Parent | on HN ]

Re:Starting from scratch: Training a 30M Topological Transfo... by tuned on Thursday January 22, 2026 @04:28AM (#46716976)

ok, thanks. I am taking it slow then

[ Parent | on HN ]