News for hackers. Stuff that matters.
Re:NanoGPT Slowrun: Language Modeling with Limited Data, Inf... by suddenlybananas on Wednesday March 04, 2026 @01:43PM (#47251946)
Reminds me a fair bit of the BabyLM challenge. It would be good to give them a shout-out and see how this challenge differs.
Re:NanoGPT Slowrun: Language Modeling with Limited Data, Inf... by sdpmas on Wednesday March 04, 2026 @01:59PM (#47252173)
hey, it's Samip (behind the Slowrun repo). yeah that's a fair point, we will mention them in the blog. but there are a couple of major differences: 1. our emphasis is on using more compute to get better data efficiency. this is important because there are lots of hacky chances that will get lower loss, but when compared to general methods that leverage a lot of compute, they don't do so well. and you can already see how this emphasis on compute leads to different methods to BabyLM! 2. our reasoning behind the repo is not anything to do with how much data a child sees. and our dataset is not tailored towards that either. it's simple pretraining on random subset of the internet. we know there are better training algorithms that get lower loss on that data, and we are finding those.
Re:NanoGPT Slowrun: Language Modeling with Limited Data, Inf... by soraki_soladead on Wednesday March 04, 2026 @02:04PM (#47252234)
also, BabyLM is more of a conference track / workshop than an open-repo competition which creates a different vibe
Re:NanoGPT Slowrun: Language Modeling with Limited Data, Inf... by Mumps on Thursday March 05, 2026 @09:54AM (#47262215)
I feel like you really need to mention BabyLM. For example you have:

> Directions we think are wide open ... Curriculum learning

BabyLM and offshoot published a pretty convincing body of work on exactly that (which suggests it's not particularly relevant to LM training).

As I read your page, I really felt like the brevity-thoroughness tradeoff went the wrong way.