Both an image and a convolution have a conjugate-symmetric DFT, and the phase of each complex bin encodes spatial information, and there's no "taking the magnitude" involved anywhere when turning a spectrum back into an image (only complex cancellation).