I tried out a 6" screen for my last phone, and when it broke, I got a Moto x4 precisely because it wasn't huge (it's still a little larger than I prefer). I have pretty big hands, but the 6" screen was never comfortable and it didn't fit nicely in my (also rather large) pockets, so the smaller phone was a better fit for me.

My main annoyance is that I'm essentially stuck with "budget" phones since all of the top of the line phones have a larger form factor. I think 5" is ideal, but it's hard to find anything under 5.5" these days, much less something with decent specs.