The architecture is something like an ensemble but there is also this control network which chooses 2 experts to generate text.