I didn't notice any issue with RNN at the lower levels of background noise, but as the background got noisier, I too heard what sounded like speaker distortion which seemed more distracting than the background noise.

I was slightly confused by the text that claimed that it was expected for the intelligibility to go down though, so maybe that's all working as intended.