Yeah, but you don't want to run an ensemble of 10 RNNs on your phone, or in the cloud for that matter, when you got billions of queries. It's too expensive.

In practice the ensemble model is compactly transferred into a single network. In order to do that, they train a new network to copy the outputs of the ensemble, exploiting "dark knowledge".

Recurrent Neural Network Training with Dark Knowledge Transfer - https://arxiv.org/abs/1505.04630v5