> The real reason we use I/Q sampling is because we want to frequency-shift a signal.
> Why do we want to frequency-shift a signal? In radio frequency applications the signal of interest almost always has a much lower bandwidth than its highest frequency. In other words, the signal has a small bandwidth (say 40 MHz) centered around a high center-frequency (say 2.4 GHz). If we want to digitize the signal, then one way would be to use a very high sample-rate ADC (e.g. a 2.4 GHz ADC). But these are very expensive, and a much better way of digitizing the signal is to use a mixer (a frequency shifter) to shift the signal to be centered around 0 Hz and then use a relatively low sample-rate ADC (e.g. a 40 MHz ADC).
> The way frequency shifting is done is by multipling the signal by a sine signal, which can be done in hardware. But this introduces a distortion to the signal because multiplying by a sine is not actually a frequency shift. It just so happens that this distortion is cancelled out by adding another copy of the signal multiplied with another sine delayed by 90°. But this addition needs to be complex (due to the relationship between sine functions and true frequency shifts), so what we do is sample the two distorted signals and do this complex addition with the digitial signals.
I'm not sure I understand you correctly, but I would not say you distort the signal when you multiply with a sine wave. Essentially you create to frequency components the sum and difference frequencies (f1+f2, f1-f2), now if f1 is your modulated signal (so some f1+fmod, where fmod is a band and can be positive and negative) and you want to convert to baseband you would select f2 so that it's at the carrier (f1=f2) then you generate a baseband signal at 0 carrier frequency and a signal at 2xf1 which is usually outside your detector bandwidth so not detected. However this process only gives you half of the frequencies of your fmod, to get the other half you need to multiply with cosine(f2) which essentially gives you the component that was at 2xf1 now at baseband. So to handle that more elegantly in math you add the two components up as real and imaginary components, essentially that enables you to drop the cos/sin(f1) terms from your equations.
You are right nobody can hear phase, but nobody can see phase either again because you need to relate (interfere) to something. However it does make a difference if we think about the superposition (interference) of different audio frequency components. We would definitely here some of those phase differences.
That said iq does not make sense in audio processing because it's baseband. There is no carrier wave.
The same is not true for radio. There, corrupting the phase relationships corrupts the data (for many systems).
Phase is relative, but our ears don't care about relative phase either (at least as long as you don't stick nonlinear filters after, then it starts mattering, but usually in audio things are fairly decorrelated anyway so it only matters in quite specific cases).
Here is an example: https://twitter.com/marcan42/status/1282685645731672064
Demo: https://twitter.com/zwegner/status/1282859889447116809 (interestingly, you can hear the change in Twitter's low quality encode, but it goes away at higher qualities, so it seems their crappy AAC encoder does care about relative phase :-))
It's counterintuitive how little our ears care about phase across frequency bands. This is not true for other kinds of signals.
However, because its virtually unknown to audio folks theres perceptible nodes everywhere, if you can hear them.
Edit, this is how the Airspy sdr works, to avoid iq imbalance like you get in the direct conversion receivers in most sdrs.
Second edit for terminology. Mixing is multiplying by a frquency to shift frequency. Baseband means you shifted the center of the frquencies you care about to zero, so half of the frequency content is negative. Negative frequencies are what drive that mean imaginary number into the whole thing.
It just so happens that audio DSP algorithms happen to almost never care about those exact kinds of processing, due to the way our ears and brains work. And thus, IQ data is not used in audio. But it's not because it's baseband. It's because our ears don't care about phase relationships (which is one thing you can more easily preserve in the IQ domain) and because frequency shifts like downconversion are not useful in music since they destroy the harmonic relationships in the sound.
100,200,400Hz is a consonant tone, while 110,210,410Hz is a dissonant mess
AM doesn't have this problem because it has symmetric sidebands and a carrier (so a tuning offset does not result in audio frequency shift), but SSB does. If you listen to an SSB transmission without your tuning being perfect, it sounds horrible. Voice sounds distorted, and music is hideous. I'm having trouble finding an example of the latter, probably because nobody dares put music through SSB :-) (but you can do this easily enough in gnuradio by upconverting a song with a 10Hz offset, for example)
Audio time stretching (or equivalently, changing the pitch without changing time) is not a clearly defined process with a mathematical description (unlike plain resampling or modulation) but rather a semi-heuristic process that takes into account psychoacoustics. But yes, in practice, for small adjustments of a monophonic sample like a voice, modern algorithms sound really good.
In the frequency domain, you could think of AM demodulation as computing the width (and phase!) between the carrier images on both sidebands. It doesn't matter if the signal is a bit off to the side, because the width will be the same. You have a mirror image which gives an absolute reference.
In the IQ domain, you look at the magnitude of the vectors, not their angle, so you don't care about the frequency.
In SSB you only have one sideband, and often no carrier at all, so there is no reference. You need to nail the frequency to get a proper signal out. And even then AIUI your phases will be random, though that doesn't matter for audio.
But nodes and mixing products are independent of overall phase across the power spectrum, in a linear system. So if you apply the same phase change to both left and right, the distribution of nodes in the room won't change. The only time these inter-frequency phase relationships start to matter is when you introduce nonlinearities, like distortion.
But typical reflections off of surfaces are largely linear as far as I know, and any linear operation will not introduce any power spectrum changes that are phase dependent. As far as I know, the the ear canal can be largely modeled as a linear system (to within the thresholds of hearability).
The only way to hear phase is to introduce a nonlinearity. That then generates harmonics (or sometimes even lower frequencies), and their power spectrum depends on the specific phase relationships of the incoming signal.
A physical example of a nonlinearity would be a vibrating surface that hits another surface at a certain excursion. Depending on the relative phases of the excitation signal, you can have different peak excursion, and therefore clearly get a different result if one phase set makes it reach the other surface and another one doesn't.
The real reason we use I/Q sampling is because we want to frequency-shift a signal.
Why do we want to frequency-shift a signal? In radio frequency applications the signal of interest almost always has a much lower bandwidth than its highest frequency. In other words, the signal has a small bandwidth (say 40 MHz) centered around a high center-frequency (say 2.4 GHz). If we want to digitize the signal, then one way would be to use a very high sample-rate ADC (e.g. a 2.4 GHz ADC). But these are very expensive, and a much better way of digitizing the signal is to use a mixer (a frequency shifter) to shift the signal to be centered around 0 Hz and then use a relatively low sample-rate ADC (e.g. a 40 MHz ADC).
The way frequency shifting is done is by multipling the signal by a sine signal, which can be done in hardware. But this introduces a distortion to the signal because multiplying by a sine is not actually a frequency shift. It just so happens that this distortion is cancelled out by adding another copy of the signal multiplied with another sine delayed by 90°. But this addition needs to be complex (due to the relationship between sine functions and true frequency shifts), so what we do is sample the two distorted signals and do this complex addition with the digitial signals.
So the reason we have complex samples is because that's the best way we've found to do frequency shifting using real-only sine waves (this explains why we don't use complex numbers in audio signal processing; there's no need to do frequency shifting!). This tutorial goes into the details and is the best explanation I've seen on quadrature sampling (another term for I/Q sampling): https://www.dsprelated.com/showarticle/192.php
I think engineers (myself included) tend to get confused because using complex numbers makes the math simpler, and so they think that's the real reason we use them. All the talk about ambiguous frequencies or negative frequencies or needing to know the phase of a sample is true, but all of those problems could be solved without complex numbers simply by sampling twice as fast and then doing some math (again, audio DSP does just fine without quadrature sampling), so it's not a "real" reason to do this strange kind of sampling.