Re:NanoChat – The best ChatGPT that $100 can buy
by ComplexSystems on Tuesday October 14, 2025 @04:37AM
(#45577635)
I haven't heard of this before. Has Muon dethroned Adam and AdamW as the standard general purpose optimizer for deep learning?
[ Parent
| on HN ]
Re:NanoChat – The best ChatGPT that $100 can buy
by spyder on Tuesday October 14, 2025 @11:55AM
(#45581557)
It's for hidden layers and not for every parameter:
From Keller's Muon github page:
"Muon is an optimizer for the hidden weights of a neural network. Other parameters, such as embeddings, classifier heads, and hidden gains/biases should be optimized using standard AdamW."
And I just looked into this nanochat repo and it's also how it's used here.
https://github.com/karpathy/nanochat/blob/dd6ff9a1cc23b38ce6...