PS you are a hero of mine - I’m an academic medical doctor for who CS231n was my first foray into AI, and since then I’ve gone on to gold medal in a couple of Kaggle competitions and secured 5 years of higher research funding to pursue clinical AI. I am immensely grateful to you and Fei-Fei Li.
https://github.com/christianversloot/machine-learning-articl...
Once you do research on the model, testing out things, it often tends to become such kwarg monster in many frameworks.
Having everything (relevant) in one file (even in the config file itself with hyper params) allows you to copy the file for every experiment and modify it inplace. This avoids the kwargs mess. But then the config files are very complex, and can become messy in other ways (esp for research projects). Example: https://github.com/rwth-i6/returnn-experiments/blob/master/2...
Such approach makes it much more flexible and does not mess with the baseline code. As you say, it's more like an evolutionary DNA-like approach, where you then tend to do crossovers with other evolved good-performing configs, etc.
The project went on to have more impact than I originally imagined and made its way into a number of projects and papers. One of those I found only a few days ago here: https://twitter.com/karpathy/status/1566100736076697600 . What I love about these projects is that the authors often "hack up" minGPT in code directly. They don't configure a comprehensive kwarg monster. I think there's a beauty in that. Very often I wish we had more gists and fewer frameworks - to look at code chunks, understand them completely, tune them to our needs, and re-use them in projects, similar to how bacteria trade little DNA plasmids. minGPT is written for those who want that for their GPT projects. There's plenty of cons to this approach too, ultimately I think there's value in both approaches.
Coming up the theme of future minGPT development: more examples, and more teeth - it should be possible to demonstrate the training of relatively serious (~few B) models with minGPT on one n-gpu node and reproduce some benchmarks around that scale, but never sacrifice its readability.