It's feasible to put the expert routing logic in a previous layer. People have done it: https://arxiv.org/abs/2507.20984