News for hackers. Stuff that matters.
Re:NanoGPT Slowrun: Language Modeling with Limited Data, Inf... by bee_rider on Wednesday March 04, 2026 @06:55PM (#47255686)
> Directions we think are wide open

> Second-order optimizers and natural gradient methods

Do second order optimizers help improve data efficiency? I assumed they’d help you get to the same minimum faster (but this is way outside my wheelhouse).

Re:NanoGPT Slowrun: Language Modeling with Limited Data, Inf... by sdpmas on Wednesday March 04, 2026 @07:35PM (#47255992)
yes! typically the optimizer that trains faster also get better data efficiency. it maybe not be absolutely true, but that has been my observation so far. also see https://arxiv.org/pdf/2510.09378 for second-order methods.
Re:NanoGPT Slowrun: Language Modeling with Limited Data, Inf... by alyxya on Wednesday March 04, 2026 @08:21PM (#47256258)
Fundamentally I don't believe second-order methods get better data efficiency by itself, but changes to the optimizer can because the convergence behavior changes. ML theory lags behind the results in practice.
Re:NanoGPT Slowrun: Language Modeling with Limited Data, Inf... by vladf on Wednesday March 04, 2026 @09:33PM (#47256769)
That still looks like a “converge faster” paper.

https://arxiv.org/abs/2006.10732

The above provides a nuanced theoretical view. GD inductive bias is probably better unless your model is misspecified