Meh, I ride the BART every day in California, and before that I rode the tube every day in London. The bigger size is just something you get used to, it doesn't prevent one-handed operation, you just use Reachability or do a little hand shimmy.

I get the elegance argument about 4/5/SE design, but the screen size thing is just the market speaking. A lot of people use their phone as primary computing device and a bigger screen is nice when it's your only device.