sdpmas
Re:NanoGPT Slowrun: 10x Data Efficiency with Infinite Compute
on Friday March 20, 2026 @01:08AM (#47450738)
oh ensemble can be distilled to a single model easily.
Re:NanoGPT Slowrun: 10x Data Efficiency with Infinite Compute
on Thursday March 19, 2026 @08:21PM (#47448517)
i think evolution meta-learns the architecture, hyperparams. some domain knowledge too (for ex, we all perceive the world as 3d) but not much. if you compare the text consumed by human vs AI (and i think this is fair b/c even with evolution text is a pretty recent invention for humans), the gap is many orders of magnitude.
Re:NanoGPT Slowrun: 10x Data Efficiency with Infinite Compute
on Thursday March 19, 2026 @07:07PM (#47447685)
the T stands for tea :)
Re:NanoGPT Slowrun: 10x Data Efficiency with Infinite Compute
on Thursday March 19, 2026 @04:03PM (#47445089)
continual learning is LLMs :)
ultimately everything will be/already is data bottlenecked.
Re:NanoGPT Slowrun: 10x Data Efficiency with Infinite Compute
on Thursday March 19, 2026 @03:45PM (#47444840)
thanks!
Re:NanoGPT Slowrun: 10x Data Efficiency with Infinite Compute
on Thursday March 19, 2026 @03:45PM (#47444836)
> you can simply generate more, and higher quality, artificial data
this is simply not true. and it's very clear if you look at continual learning, robotics, biology, etc. each has enough economic incentives to spend 1000x compute if that led to much better results, but we just don't know how to do that.
good point on chinchilla, but our models are still absurdly large no matter what standards you compare them to.
Re:NanoGPT Slowrun: Language Modeling with Limited Data, Infinite Compute
on Thursday March 05, 2026 @03:50AM (#47259260)
absolutely!
Re:NanoGPT Slowrun: Language Modeling with Limited Data, Infinite Compute
on Wednesday March 04, 2026 @07:55PM (#47256110)
thanks, here's another one: https://arxiv.org/abs/2511.03276
Re:NanoGPT Slowrun: Language Modeling with Limited Data, Infinite Compute
on Wednesday March 04, 2026 @07:37PM (#47255998)
diffusion is promising, but still an open question how much data efficient they are compared to AR. in practice, you can also train AR forever with high enough regularization, so let's see.
Re:NanoGPT Slowrun: Language Modeling with Limited Data, Infinite Compute
on Wednesday March 04, 2026 @07:35PM (#47255992)
yes! typically the optimizer that trains faster also get better data efficiency. it maybe not be absolutely true, but that has been my observation so far. also see https://arxiv.org/pdf/2510.09378 for second-order methods.
Re:NanoGPT Slowrun: Language Modeling with Limited Data, Infinite Compute
on Wednesday March 04, 2026 @07:33PM (#47255977)
no ensembling means train 8 models and during inference avg logits of all 8 models to make a prediction.
Re:NanoGPT Slowrun: Language Modeling with Limited Data, Infinite Compute
on Wednesday March 04, 2026 @05:33PM (#47254929)
yeah, we do incorporate some of the findings from the paper in our repo! like aggressive regularization and ensembling.
Re:NanoGPT Slowrun: Language Modeling with Limited Data, Infinite Compute
on Wednesday March 04, 2026 @05:32PM (#47254918)
yes, agreed, modded-nanogpt is already a data-efficient variant of original nanogpt. just that the kinds of algorithms it allows are somewhat constrained because it optimizes for wall clock time.
Re:NanoGPT Slowrun: Language Modeling with Limited Data, Infinite Compute
on Wednesday March 04, 2026 @02:31PM (#47252601)
yes, good point. right now, it's somewhat hard to overfit because the meta-optimization extracts tiny bits of information. but over time, we will switch the validation set to some other random subset of the FineWeb or even entirely OOD datasets!
Re:NanoGPT Slowrun: Language Modeling with Limited Data, Infinite Compute
on Wednesday March 04, 2026 @01:59PM (#47252173)
hey, it's Samip (behind the Slowrun repo). yeah that's a fair point, we will mention them in the blog. but there are a couple of major differences:
1. our emphasis is on using more compute to get better data efficiency. this is important because there are lots of hacky chances that will get lower loss, but when compared to general methods that leverage a lot of compute, they don't do so well. and you can already see how this emphasis on compute leads to different methods to BabyLM!
2. our reasoning behind the repo is not anything to do with how much data a child sees. and our dataset is not tailored towards that either. it's simple pretraining on random subset of the internet. we know there are better training algorithms that get lower loss on that data, and we are finding those.