People have tried that, but the model essentially learns to discard the phase channel because it is too hard for it to learn any useful information from it.