Second partially true (as validated by intersubjective tests) guess is that the harmonic structure matters. It is a major component of the coloration, and funnily enough low non-tapering one is discernible from low tapering one despite what SINAD/THD graphs suggest. Amount is discernible too, especially of second and third harmonic, and total of higher harmonics, but we have not established the thresholds. Intermodulating injected ultrasound medium to high level garbage (as produced by some poorly implemented power supplies in bad PSRR amps) is sometimes detectable and tests tend to miss it. It also sounds different from just bad SINAD/IMD.
AES researchers (Earl Geddes, Lidia Lee among others) tried for a better harmonic distortion metric in the past. It went nowhere because industry always wins.
And if the amplifier does outright clip, either in current or in voltage, that sounds just rough.
You arent going to hear intermodulation distortion at -110dBfs regardless of how you weight it.
I want zero distortion or as close to it, not euphonic distortion unless I put it in there via equalization or effects or whatever.
Power delivery is a secondary parameter for qualitative analysis. Although for end user it may be actually the main parameter and SINAD is a secondary.
For DACs, I think SINAD (for those unfamiliar, this is equivalent to THD+N) really does tell something very close to the whole story. Call it 95%, 99%, 99.9%, whatever you will. The remaining N% is stuff we probably don't yet have reliable measurements for. I'm not saying it's magic and unmeasurable stuff, just that we don't have a solid set of measurements to capture it.
For headphone amps, or DAC+amp combos, there are definitely things not well represented by SINAD. I think most of this is related to power delivery (which he does measure) and I think the rest is probably related to slew rate or something thereabouts. But that's just a wild-ass psuedo scientific guess on my part.