Re:Qwen 3.8 27B is excellent, but it defaults to overthinkin...
by rao-v on Monday August 17, 2026 @01:20PM
(#49334481)
Even Muse Glimmer (as did GPT-OSS I think) does ~4 sliding window attention layers + 1 full attention layer (like Gemma 4). I’m assuming both labs have good reason to think that gated delta nets are not optimal.
Of course it’s possible the labs just stick with the optimal architecture for large models and GDN is best for smaller models.
Yes, this is something I hope they will change. Gemma 4 31B is much slower on pre-Blackwell GPUs as a result, which is a bit of a shame for local model experimentation.