You could double up on gains on a smaller phone by reducing the screen resolution, which reduces the power demand on the chip managing the display, and so on. Back when I had the Nexus 4, at 720p, I never felt it was pixelated or annoying. Then again, I don't watch movies on my phone either.
I think the dominant market pressures so far has been practical concerns like phone thickness and battery life so manufacturers - who are all also competing with their past products - are forced to sacrifice screen size options to compete. Now screen size is becoming unwieldy and screens/batteries have steadily improved so we might swing back the other way.