News for hackers. Stuff that matters.
Re:GPT-4o by og_kalu on Tuesday May 14, 2024 @02:07AM (#40352113)
A Mixture of Experts model is still one behemoth neural network and believing otherwise is just a common misconception on term.

MoE are attempts at sparsity, only activating a set number of neurons/weights at a time. They're not separate models stitched together. They're not an Ensemble. I blame the name at this point.

There are no comments on this comment yet.