Phase shifts make no practical difference when you're equalizing headphones. You really shouldn't need to use notch filters there.

EQ phase is relevant if the dry signal has a chance to mix with the EQ output.

Changes in phase is not "distortion". When you end up making large boosts (especially to make sub bass audible), that's when your headphone drivers might start to distort sound. This is fixed with less volume, obviously.

An EQ that cuts frequencies can actually increase the amplitude of the waveform due to phase shifts. It's not just boosting. What I'm talking about is more relevant on analog gear which is more sensitive to volume.

Analog distortion of signals also affects the phase. A digital filter might or might not compensate for it, but it's not always bad.

Also, human ears tend to be more sensitive to amplitude than to the phase. Especially if the same filter is applied on both channels, which leaves the phase difference unchanged.

It's more than just human ear sensitivity. Phase shifting changes the relationship between frequencies in a sound.

For example, phase shifting can change the amplitude of a sound. Analog hardware is especially sensitive to changes in amplitude so you might be introducing distortion just by shifting phase.

Depends largely on how it's implemented. There is a big difference between the quality of a well implemented convolution engine and a crappy biquad filter.

Where would something like Fabfilter fall in the scale? I imagine there's some pretty complex stuff going on under the hood.

Fabfilter isn't really doing anything magical. Its default mode is a pretty normal EQ with phase shifting. Linear-phase mode is like any other linear-phase mode with the same drawbacks.

Probably the UI that made me think it was a Wonka creation. It's really not sonically much different to a stock EQ.

Note that not all biquad filters are crappy.

The primary measurement is called group delay, which is the derivative of phase with respect to frequency (since a linear phase is just a uniform delay). Fortunately we have studies from the 70s publicly available that establish a group delay audibility threshold.

FIRs add vanishingly small amounts of distortion. Their real drawback is in the added delay. For asynchronous music playback this is nothing to worry about, but for video or anything interactive (communications or gaming) it's going to be a real tough pill to swallow.

At headphone sizes, does your DAC have a good enough clock to even speak about phasing?