News for hackers. Stuff that matters.
Re:NanoGPT Slowrun: Language Modeling with Limited Data, Inf... by jiggawatts on Thursday March 05, 2026 @02:58AM (#47258889)
That doesn't seem all that different to a MoE architecture.
Re:NanoGPT Slowrun: Language Modeling with Limited Data, Inf... by yorwba on Thursday March 05, 2026 @03:47AM (#47259239)
It's the opposite of a MoE architecture in many ways. MoE splits every individual feed-forward layer into many tiny subnetworks, only a small number of which contribute to the layer output, and they get trained together to complement each other.

Ensembling makes multiple copies of the entire model, trains them independently on the same task, and then has every copy contribute to the output.

Reducing computation vs. increasing it; operating at per-layer granularity vs. whole model; specialization vs. redundancy.