News for hackers. Stuff that matters.
Re:NanoGPT Slowrun: Language Modeling with Limited Data, Inf... by sdpmas on Wednesday March 04, 2026 @02:31PM (#47252601)
yes, good point. right now, it's somewhat hard to overfit because the meta-optimization extracts tiny bits of information. but over time, we will switch the validation set to some other random subset of the FineWeb or even entirely OOD datasets!

There are no comments on this comment yet.