With respect to recorded music it tends to be the opposite: if you pitch up a track then the rhythm feels tighter and the vocals more steady, because the differences between the intent and the performance are relatively smaller. Hence an old studio trick dating at least back to the Chipmunks was to record in a lower key. Listener discrimination seems relatively less important in practice, on the other hand.