News for hackers. Stuff that matters.
Re:NanoGPT Slowrun: Language Modeling with Limited Data, Inf... by sdpmas on Wednesday March 04, 2026 @07:37PM (#47255998)
diffusion is promising, but still an open question how much data efficient they are compared to AR. in practice, you can also train AR forever with high enough regularization, so let's see.
Re:NanoGPT Slowrun: Language Modeling with Limited Data, Inf... by _0ffh on Wednesday March 04, 2026 @07:39PM (#47256008)
Yes, it could go either way of course.

Still, just for reference, here's the paper I remembered: https://arxiv.org/pdf/2507.15857

Re:NanoGPT Slowrun: Language Modeling with Limited Data, Inf... by sdpmas on Wednesday March 04, 2026 @07:55PM (#47256110)
thanks, here's another one: https://arxiv.org/abs/2511.03276