Instead of using fancy deep learning to fake the voice why not simulate a degraded signal by adding noise and using a good text to speech program (after you have figured out gender ).