News for hackers. Stuff that matters.
Re:Is One Layer Enough? A Single Transformer Layer Matches F... by kmeisthax on Saturday July 04, 2026 @12:48AM (#48782707)
No. Mixture-of-Experts is for dynamically loading different sets of weights[0] per-token. The idea with MoE is that you can have more weights than fit in RAM without having to actually load everything in RAM.

My idea is more about creating recurrent paths in an otherwise all-forward network. The same number of weights would be loaded, we'd just be routing residuals differently.

[0] Usually multi-layer perceptron / linear portion weights - although maybe someone's tried attention head MOE?

There are no comments on this comment yet.