100,200,400Hz is a consonant tone, while 110,210,410Hz is a dissonant mess
AM doesn't have this problem because it has symmetric sidebands and a carrier (so a tuning offset does not result in audio frequency shift), but SSB does. If you listen to an SSB transmission without your tuning being perfect, it sounds horrible. Voice sounds distorted, and music is hideous. I'm having trouble finding an example of the latter, probably because nobody dares put music through SSB :-) (but you can do this easily enough in gnuradio by upconverting a song with a 10Hz offset, for example)
Audio time stretching (or equivalently, changing the pitch without changing time) is not a clearly defined process with a mathematical description (unlike plain resampling or modulation) but rather a semi-heuristic process that takes into account psychoacoustics. But yes, in practice, for small adjustments of a monophonic sample like a voice, modern algorithms sound really good.
In the frequency domain, you could think of AM demodulation as computing the width (and phase!) between the carrier images on both sidebands. It doesn't matter if the signal is a bit off to the side, because the width will be the same. You have a mirror image which gives an absolute reference.
In the IQ domain, you look at the magnitude of the vectors, not their angle, so you don't care about the frequency.
In SSB you only have one sideband, and often no carrier at all, so there is no reference. You need to nail the frequency to get a proper signal out. And even then AIUI your phases will be random, though that doesn't matter for audio.