Maybe I've just grown accustomed to it but I never felt like it was huge, as I did the first time I went from an older phone with a 3" diag screen to a 4.3" one.
I think a big part of it is the fact that many users spend most of their time browsing the web (easier on the eyes with a big screen), using mobile apps and games (same thing), and typing messages (easier to use onscreen keyboard with two thumbs when the buttons aren't tiny). The bigger phones look and feel weird when you're holding this big slab up to your head but I don't think I've used mine to make a phone call in a month now.
I don't wear baggy pants but these 5-5.5" phones always fit in my pockets. Even in slim jeans I have no issue with the Moto X and I think some of that has to do with the curved back. The flat screen sits against my leg and the curve means there aren't any sharp edges sticking out.
Other that not needing a new phone yet, it's something that's kept me from buying one of the newer Nexus 6P phones. I really like the feature set and it looks nice enough but it seems like it's maybe at a point where that pocketability will suffer.
Also, kudos to Motorola for creating and supporting a great phone. My previous phones have all slowed down to the point of being unusable over a couple years of system updates, but the Moto X still feels snappy.
Ever since I started carrying a mobile phone I stopped wearing a watch. I don't need or want both.
I can understand why it's happening, though. Netbooks disrupted laptops while the smartphone entered as a new market. Tablets disrupted netbooks. Now smartphones are moving up-market, with phablets, and disrupting tablet sales. That's a large existing market for the taking, which will give much better revenue growth than trying to carve out a niche in the small smartphone segment.
I think we'll see smartphones continue growing into the 6-7" screensize range, so that they're sized for a purse or (barely) a large pocket. And then smartwatches will disrupt the low end, so that you don't need to take your phone out of your purse/pocket as often. It's already happening - many smartwatch models let you take calls & do anything voice-related on the watch, so you only use your phone as an input device when you need to type.