Yes, mathematically, you can get impulse response via Fourier transform. In practice, you're using a current to push a coil against a magnet into a spring. A real current, a real magnet, a real spring. You get all sorts of non-linear behaviors. On top of that, the spring (that is, the driver) retains impulse energy as mechanical energy, and then emits it back in a nonlinear way, by moving that coil in the magnetic field - turning a motor into a generator, producing seriously nonlinear signal back into the amplifier that produced the original signal, where it's picked up by the distortion-correcting negative feedback loop, and...
Yeah, that's not a simple Fourier transform anymore. If you had enough data points, it could be modeled mathematically. We don't have all the data, and don't yet have a good way to measure all that stuff.
For example, consider a system that just squares or cubes its input. The unit impulse response will look the same in both cases. The frequency response isn't even defined, since the steady state response to a sinusoid is not itself a sinusoid.
Microphones really struggle with this. So do mic preamps, recording media, amplifiers, and especially speakers. Recorded instruments sound wildly different from live-in-a-room instruments.
edit: I should add here that fast impulse response isn't necessarily what we want. Consider the common use of the relatively sluggish Shure SM57 on snare drums, rather than the much faster response of a small diaphragm condenser. The condenser usually just sounds harsh. The SM57 smooths out the sound. This is good, because nobody in their right mind actually sticks their ear one inch from a snare drum.
"Recorded instruments sound wildly different from live-in-a-room instruments." Yes, they do.
In the best case, you have two nearly perfectly accurate transducers located inside the not-quite-ear-canals of a dummy head doing the recording, and you play those two signals back to transducers located similarly in the head of your listener. This is the best case for accurate reproduction of instruments. You can get a recording that sounds nearly the same as being in the room where it happened.
In the most common "purist" method, recording engineers are using two or three mics arrayed in free space somewhere near where the audience would be, and hoping to capture the sensation of being in the room. This is problematic because the instruments are complex, moving three-dimensional shapes producing sound in a field which interacts with all sorts of things before being sampled by those 2-3 points. That throws away nearly all the available information. Then playback emanates from two not-really-point sources in a completely different room.
In the normal non-purist methods, engineers close-mike each of the instruments, take direct input from some electronic instruments, and may or may not try to capture some room ambience. Then they process everything to the point where they are as much a performer as the recorded players, and work on it over and over again until (hopefully) everyone is happy with it as a work of art. There should be no pretense that this is going to get you an accurate rendition of the feeling of being there, though.
I don't really care if a recording is accurate. I care if it's enjoyable.
When I think of "the feeling of being there", I don't think of the feeling of being in the room with the musicians. I think of the feeling of what you're doing when you experience that music - where does it take you? I'll never be able to separate Pink Floyd's The Wall from making out with the girlfriend who introduced me to Pink Floyd, for example. Music has strong sense-memory, almost as strong as smell. That's what I want to engage - I want to make music you remember.
It's true. It should be added, however, that when the frequency response is complex (and it always is with transducers) and the measurement conditions are not very repeatable, it's probably beneficial to also get an impulse response.
> Phase shift can also be measured through this approach
I thought headphones are zero-phase-shift devices? This is because of the very small volume of air between the membrane and the eardrum - they both move in lockstep, hence zero phase shift.
Audio engineers routinely choose transient response over frequency response, hence the popularity of Yamaha NS10 and Auratone 5C monitors in professional environments.
Sure, nonlinear distortions are not covered and also sure that the measurements won't cover everything which will influence the perceived sound. But nevertheless frequency response measurements are a very solid tool which allows objective comparions between various systems.