Two things come to mind though: voltage/current delivery of your amp, and damping ratio.
The first depends on the characteristics of your amplifier. Some are better at delivering current, some voltage. The lower impedance (32) is better suited to high-current/low-voltage sources, which includes most portable devices, phones, etc. Conversely, the higher impedance (250) is better for high-voltage/low-current sources like tube amps.
The second is about the ratio of the headphone impedance to the amp output impedance. You want a high ratio, so if your source has a large output impedance then the higher impedance headphones will sound better. Good headphone amps sometimes specify the output impedance, or you can measure it.
Headphone outputs of mixers fall into this category as well. Proper audio interfaces and sound cards have no issues at all driving 250 Ω to deafening volumes. Laptops no issues as well (for me).