I was so confused by the saltiness until I saw the username. I'm sure you've earned it.

I got into deep learning because of your char-rnn posts a while ago -- it inspired me to do an undergrad thesis on the topic. I read arxiv papers after that and implemented things from the ground up until a startup liked my work and hired me in a neural network engineer position.

Fast forward a few years and I was enamoured with minGPT and it stuck with me. I wanted a CIFAR10 experimentation toolbench so I took my hand at my best swing at the minGPT treatment on the current best single GPU Dawnbench entry, added a few tweaks and got https://github.com/tysam-code/hlb-CIFAR10. It currently (AFAIK) holds the world record for training to the 94% mark by a fair bit.

It's about 600 lines in a monolithic file, only requiring torch and torchvision, but it's my first project like this and I'd like to learn how to better minify codebases like this. It seems like the hardest part is knowing how to structure inheritance and abstraction, but I don't know if you had any good outside references/resources that you used or would recommend.

I'm hoping to apply the Dawnbench treatment to a small language model at some point, pick a validation loss watermark or some reasonable metric, then optimize around that obsessively to build a good tiny reference model. I don't know if you'd know anyone that's interested in that kind of thing, but I feel like that would be a fun next step for me.