I expect that's true. After I found the Nexus 4 too big, and really didn't want to go for the 5 or 6, it took me quite a long time to find a high-end smaller phone (ended up with the Sony Z3 Compact... where "compact" these days apparently means 4.6"). Most manufacturers on the Android side seem to equate top-of-the-line with having a large screen.

For me, I think the ideal screen size is around 4.3". But good luck finding a high-end phone with that screen size.