Re:NanoGPT Slowrun: Language Modeling with Limited Data, Inf...
by sdpmas on Wednesday March 04, 2026 @02:31PM
(#47252601)
yes, good point. right now, it's somewhat hard to overfit because the meta-optimization extracts tiny bits of information. but over time, we will switch the validation set to some other random subset of the FineWeb or even entirely OOD datasets!
[ Parent
| on HN ]
There are no comments on this comment yet.